Storage medium storing information processing program, information processing device, information processing system, and panoramic video display method

ABSTRACT

An example information processing device displays a partial area of a panoramic video on a display device. The information processing device obtains range information with which it is possible to identify a display range of the panoramic video to be displayed and/or to identify a position within the display range. The obtained range information is information regarding a display range determined based on an input made on a predetermined input device while the panoramic video is played. Where the panoramic video is output using the display device, the information processing device outputs the panoramic video while changing output content thereof in accordance with the obtained range information.

CROSS REFERENCE TO RELATED APPLICATION

The disclosure of Japanese Patent Application No. 2012-125644 filed onJun. 1, 2012 is herein incorporated by reference.

FIELD

The technology herein relates to a storage medium storing an informationprocessing program, an information processing device, an informationprocessing system, and a panoramic video display method for displaying apanoramic video on a display device.

BACKGROUND AND SUMMARY

There are conventional techniques for producing and displaying apanoramic image on a display device.

Conventional techniques merely display a panoramic image, and it hasbeen desired to provide various information together with the panoramicimage.

Thus, the present disclosure provides a storage medium storing aninformation processing program, an information processing device, aninformation processing system, and a video display method capable ofpresenting more information to the user when displaying a panoramicimage.

(1)

An example storage medium described in the present specification storesan information processing program to be executed by a computer of aninformation processing device for displaying a partial area of apanoramic video on a display device. The information processing programinstructs the computer to execute: obtaining range information; andoutputting at least a part of the panoramic video.

The range information is information with which it is possible toidentify a display range of the panoramic video to be displayed and/orto identify a position within the display range, and is informationregarding a display range determined based on an input made on apredetermined input device while the panoramic video is played. Theinformation processing device outputs at least a part of the panoramicvideo while changing output content thereof in accordance with theobtained range information.

With configuration (1) above, range information is obtained when apanoramic video is played. Then, at least a part of the panoramic videois output while changing the output content thereof in accordance withthe obtained range information. Thus, the playback content from pastinstances of playback of the panoramic video or other instances ofplayback of the panoramic video on other devices can be presented to auser. For example, the playback content from past instances of playbackof the panoramic video or other instances of playback of the panoramicvideo on other devices can be reflected in the output of the panoramicvideo currently played. Thus, with configuration (1) above, it ispossible to present more information to a user while displaying apanoramic image.

(2)

The obtained range information may be stored in a storage unit ashistory information. Then, information based on the history informationis output after the history information is stored.

With configuration (2) above, range information is stored as historyinformation while displaying a panoramic image. Then, the panoramicvideo is output while changing the output content thereof in accordancewith the obtained and stored history information. Thus, it is possibleto output in accordance with the history information while playing apanoramic video. That is, it is possible not only to simply present apanoramic video, but also to present information interesting to a usertogether with a panoramic video.

(3)

The history information stored in the storage unit may be information inwhich time information is associated with range information, wherein itis possible with the time information to identify a playback time pointin the panoramic video, and wherein it is possible with the rangeinformation to identify a display range at the playback time pointand/or a position within the display range.

With configuration (3) above, by storing the time information and therange information, it is possible to store ranges of a panoramic imagethat have been looked at. Thus, for example, it is possible to latercheck which range or ranges of the image area of a panoramic video havebeen looked at.

(4)

The information processing program may instruct the computer to furtherexecute determining the display range based on an input made on apredetermined input device. Then, a panoramic video of a display rangedetermined based on the input made on the predetermined input device isdisplayed on the display device, and history-related informationproduced based on the history information is output.

With configuration (4) above, history-related information can bepresented to a user while a panoramic video is played, and the user canknow information regarding a history of another user.

(5)

The history-related information output together with the panoramic videomay represent a direction of the display range identified by the rangeinformation included in the history information with respect to acurrent display range.

With configuration (5) above, as the history-related informationrepresenting a direction is presented, it is possible to present, to auser, where in a panoramic video another user was seeing, with respectto the current display range.

(6)

The output history-related information may represent a display frequencyfor a predetermined range in the panoramic video obtained from thehistory information.

With configuration (6) above, it is possible to present which locationsin a panoramic video are much seen. Therefore, for example, a user canplay a panoramic video while setting the display range to a range thatis popular (much seen by other users) or while setting the display rangeto a range that is not much seen by other users.

(7)

The history-related information may represent a display frequencydetermined in accordance with the number of pieces of historyinformation that satisfy a predetermined condition regarding the timeinformation and the range information.

With configuration (7) above, it is possible to present which locationsin a panoramic video have been much seen by other users over a pluralityof instances of playback of a panoramic video.

(8)

The history-related information may represent a display frequencydetermined in accordance with the number of pieces of range informationthat satisfy a predetermined condition regarding a position on apanoramic image.

With configuration (8) above, it is possible to present which locationsin a panoramic video have been much seen by other users over a singleinstance of a plurality of instances of playback of the panoramic video.

(9)

The panoramic video may be stored in the storage unit, which can beaccessed by the information processing device, together with objectinformation representing a position and/or a range of an objectappearing in the panoramic video. Then, history-related informationoutput together with the panoramic video may represent a displayfrequency of the object calculated based on the object information andthe history information.

With configuration (9) above, it is possible to present, to a user, howmuch attention an object appearing in a panoramic video has beendrawing.

(10)

When playback of the panoramic video is completed, a panoramic image ofa display range determined based on the range information stored as thehistory information during the playback is displayed on the displaydevice as a still image or a video.

With configuration (10) above, after completion of playback of apanoramic video, it is possible to present a history of the playback toa user.

(11)

Range information representing the display range and/or a positionwithin the display range may be obtained as the history information.Then, the information processing program instructs the computer tofurther execute determining whether or not an object appearing in thepanoramic video is included in a display range identified by the historyinformation so as to select an object to be output based on a result ofthe determination. A panoramic image including the selected objecttherein is displayed on the display device as a still image or a video.

With configuration (11) above, after completion of playback of apanoramic video, it is possible to present, to a user, the degree bywhich an object appearing in the panoramic video has been displayed(whether it has been much seen or not).

(12)

The information processing program may instruct the computer to furtherexecute determining whether or not an object is included within adisplay range during playback of the panoramic video so as to select anobject to be output based on a result of the determination. Then,information representing the selected object is obtained as the rangeinformation. A panoramic image of a display range including therein theobject represented by the range information is displayed on the displaydevice as a still image or a video.

With configuration (12) above, as with configuration (11), aftercompletion of playback of a panoramic video, it is possible to present,to a user, the degree by which an object appearing in the panoramicvideo has been displayed.

(13)

The history information stored in the storage unit may be information inwhich time information is associated with range information, wherein itis possible with the time information to identify a playback time pointin the panoramic video, and wherein it is possible with the rangeinformation to identify a display range at the playback time pointand/or a position within the display range. Then, a panoramic image of adisplay range determined based on a plurality of pieces of historyinformation is displayed on the display device as a still image or avideo.

With configuration (13) above, after completion of playback of apanoramic video, it is possible to present, to a user, which locationsof the panoramic video have been much seen (or not seen), etc.

(14)

Range information regarding a panoramic video played on the displaydevice may be obtained.

With configuration (14) above, it is possible to obtain rangeinformation from a past instance of playback on a display device onwhich a panoramic video is played while reflecting history information.Thus, for example, a user can check the playback content from a pastinstance of playback on the same display device.

(15)

Range information regarding a panoramic video played on another displaydevice different from the display device may be obtained.

With configuration (15) above, it is possible to present the playbackcontent on a display device different from the display device on which apanoramic video was played.

(16)

The information processing program may instruct the computer to furtherexecute: determining the display range; and displaying the panoramicvideo of the determined display range on the other display device. Thedisplay range is determined based on an input made on the predeterminedinput device while the panoramic video is displayed on the displaydevice. The panoramic video of the determined display range is displayedon the other display device while the panoramic video is displayed onthe display device.

With configuration (16) above, a user can view the area of an intendeddisplay range in a panoramic video, and view the panoramic video in amanner of output such that the history information is reflected.

(17)

The information processing program may instruct the computer to furtherexecute outputting the panoramic video on another display devicedifferent from the display device while the panoramic video is displayedon the display device by means of a first output control unit. Then, therange information is obtained when the panoramic video is played on theother display device. The panoramic video is displayed on the displaydevice while changing output content thereof in accordance with therange information while the panoramic video is played on the otherdisplay device.

With configuration (17) above, a panoramic video is displayed on twodisplay devices, wherein the playback content from one of the displaydevices is reflected in the output on the other display device.Therefore, where a plurality of users are seeing a single panoramicvideo at the same time, it is possible to present, together with thepanoramic video, information interesting to a user, i.e., whichlocations of the panoramic video are seen by other users.

(18)

The information processing program may instruct the computer to furtherexecute determining the display range based on an input on thepredetermined input device while the panoramic video is displayed on thedisplay device by means of a first output control unit. Then, apanoramic video of the determined display range is displayed on theother display device. A panoramic video of a predetermined range isdisplayed on the display device, and a panoramic video of a displayrange based on the range information is displayed on the display device.

With configuration (18) above, where a panoramic video is displayed ontwo display devices, a panoramic image of a display range on one of thedisplay devices is also displayed on the other display device. Thus,also the user seeing the other display device can easily check theplayback content from one of the display devices. For example,configuration (18) above is advantageous in cases where one of thedisplay devices is a portable type (hand-held type) display device, andin similar cases.

(19)

A panoramic video of a display range determined based on the rangeinformation may be displayed on the display device.

With configuration (19) above, it is possible to play a panoramic videowith the same display range as that from a past instance of playback(when history information was stored).

(20)

Range information regarding a panoramic video played on another displaydevice different from the display device may be obtained. Then, thepanoramic video is displayed on the display device while reducing achange in the display range determined based on the range information.

With configuration (20) above, it is possible to provide a panoramicvideo of the display range in an easy-to-view manner for a user who isnot controlling the display range.

(21)

The obtained range information may represent a position specified by auser while the panoramic video is played. Then, an image representing aposition on a panoramic image determined based on the range informationis displayed on the display device together with the panoramic video.

With configuration (21) above, a position specified by a user during aseparate instance of playback of a panoramic video can be presented toanother user.

(22)

An image representing a viewing direction corresponding to a displayrange or a position based on the range information may be displayed onthe display device.

With configuration (22) above, it is possible to present, to a user,which direction another user was facing during a separate instance ofplayback of a panoramic video.

(23)

An image representing a space represented by the panoramic video and aposition in the space corresponding to a position on the panoramic videobased on the range information may be displayed on the display device.

With configuration (23) above, the position represented by the rangeinformation can be expressed by an image as viewed from a differentviewpoint from the panoramic video, and it is possible to present theposition to a user in an easier-to-understand manner.

(24)

The range information may be obtained from an external device via anetwork.

With configuration (24) above, it is possible to present, to a user,information based on history information of another user.

(25)

User information representing a user who has made an input regarding therange information may be further obtained. Then, the user information isoutput when the panoramic video is output.

With configuration (25) above, it is possible to present, to a user,information indicating who is the user who was playing the panoramicvideo when the history information was stored.

Note that the present specification discloses an information processingdevice and an information processing system having units equivalent tothose realized by executing the information processing program ofconfigurations (1) to (25) above. The present specification alsodiscloses a panoramic video display method to be carried out inconfigurations (1) to (25) above.

With the storage medium storing an information processing program, theinformation processing device, the information processing system, andthe panoramic video display method described above, it is possible topresent more information to a user when displaying a panoramic image byoutputting a panoramic video while changing the output content thereofin accordance with range information that has been obtained in aseparate instance of playback of the panoramic video.

These and other objects, features, aspects and advantages will becomemore apparent from the following detailed description when taken inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a non-limiting example informationprocessing system according to the present embodiment;

FIG. 2 shows an example of a panoramic video (image) displayed on aninformation processing system;

FIG. 3 shows an example of a method for determining a display area inaccordance with an attitude of a terminal device;

FIG. 4 shows an example of an operation by a user of inputting acomment;

FIG. 5 shows an example of a panoramic video displaying a button forinputting additional information;

FIG. 6 shows an example of information (referred to as “inputinformation”) stored in response to an input of additional information;

FIG. 7 shows an example of a method for determining a conditionregarding range information;

FIG. 8 shows another example of a method for determining a conditionregarding range information;

FIG. 9 shows an example of a comment output period;

FIG. 10 shows an example of an image in which additional information isdisplayed together with a panoramic video;

FIG. 11 shows another example of additional information to be output;

FIG. 12 shows an example of object information;

FIG. 13 shows another example of an image in which additionalinformation is displayed together with a panoramic video;

FIG. 14 shows another example of an image in which additionalinformation is displayed together with a panoramic video;

FIG. 15 shows another example of an image in which additionalinformation is displayed together with a panoramic video;

FIG. 16 shows an example of a positional relationship between a displayrange and an additional information input position;

FIG. 17 shows an example of guide information;

FIG. 18 shows an example of an image displayed on each display devicewhere a target is set on another display device;

FIG. 19 shows an example of an image displayed on each display devicewhere a display range is set as a target;

FIG. 20 shows an example of history information;

FIG. 21 shows an example of an image of history-related information;

FIG. 22 shows another example of an image of history-relatedinformation;

FIG. 23 shows another example of an image of history-related informationbased on history information;

FIG. 24 shows an example case where a panoramic video of a display rangedetermined based on history information is displayed on a monitor 4;

FIG. 25 shows an example case where an auxiliary image is displayedtogether with a panoramic video;

FIG. 26 shows an example of an image representing a viewing direction;

FIG. 27 shows an example case where an image representing a positionbased on range information is displayed together with a panoramic image;

FIG. 28 shows an example of an image displayed after the completion ofplayback of a panoramic video;

FIG. 29 shows an example of data stored in a storage unit of aninformation processing system in the present embodiment;

FIG. 30 is a flow chart showing an example of a flow of a video playbackprocess in a first mode to be performed by a CPU of an informationprocessing device in the present embodiment;

FIG. 31 is a flow chart showing an example of a flow of a video playbackprocess in a second mode to be performed by a CPU of an informationprocessing device in the present embodiment;

FIG. 32 is a flow chart showing an example of a flow of a video playbackprocess in a third mode to be performed by a CPU of an informationprocessing device in the present embodiment; and

FIG. 33 is a flow chart showing an example of a flow of a video playbackprocess in a third mode to be performed by a CPU of an informationprocessing device in the present embodiment.

DETAILED DESCRIPTION OF NON-LIMITING EXAMPLE EMBODIMENTS

[1. Configuration of Information Processing System]

An information processing system, an information processing device, aninformation processing program and a video display method according toan example embodiment will now be described. FIG. 1 is a block diagramshowing an example information processing system according to thepresent embodiment. In FIG. 1, an information processing system 1includes a terminal device 2, an information processing device 3, amonitor 4, and a controller device 7. The information processing system1 of the present embodiment produces and displays an image (panoramicimage) on a display device (the terminal device 2 and/or the monitor 4).

In the information processing system 1, in response to an input made onthe terminal device 2 (and the controller device 7), the informationprocessing device 3 performs an information process (a video playbackprocess shown in FIGS. 30 to 33), and an image obtained as a result ofperforming the process is displayed on the terminal device 2 and/or themonitor 4. Thus, in the present embodiment, the information processingsystem 1 has its input function, information processing function anddisplay function implemented by a plurality of devices. Note that inother embodiments, the information processing system 1 may be a singleinformation processing device (e.g., a hand-held-type or portable-typeinformation processing device) having these functions.

The terminal device 2 is an input device that can be held by a user.That is, the terminal device 2 is a portable-type input device. Theterminal device 2 is capable of communicating with the informationprocessing device 3. The terminal device 2 transmits, to the informationprocessing device 3, operation data representing an operation performedon the terminal device 2. In the present embodiment, the terminal device2 includes a display unit (an LCD 11), and the terminal device 2 is alsoa display device. Where an image is transmitted from the informationprocessing device 3, the terminal device 2 displays the image on thedisplay unit. The terminal device 2 also includes a speaker. Where asound is transmitted from the information processing device 3, theterminal device 2 outputs the sound from the speaker.

The terminal device 2 includes a touch panel 12 as an input unit. Thetouch panel 12 is an example of a position detection unit for detectinga position at which an input has been made on a predetermined inputsurface (the screen of the display unit) provided on a housing 10.Moreover, as an input unit, the terminal device 2 includes an operationunit 13 such as a slide pad, a cross-shaped key, buttons, etc.

The terminal device 2 includes an acceleration sensor 14 as an inputunit. The acceleration sensor 14 detects the acceleration for apredetermined axis direction (although there are three axis directionsin the present embodiment, there may be one or more axis) of theterminal device 2. The terminal device 2 includes a gyrosensor 15 as aninput unit. The gyrosensor 15 detects the angular velocity with respectto the rotation about a predetermined axis direction (although there arethree axis directions in the present embodiment, there may be one ormore axis) of the terminal device 2. The acceleration sensor 14 and thegyrosensor 15 are sensors for detecting information used for calculatingthe attitude of the terminal device 2 (information with which theattitude can be calculated or estimated). Note that in otherembodiments, the attitude of the terminal device 2 may be calculated byany method. For example, the attitude of the terminal device 2 may becalculated by using another sensor other than the sensor described aboveor by using a camera capable of capturing an image of the terminaldevice 2.

The information processing device 3 performs an information process tobe performed on the information processing system 1, such as a processof producing an image. In the present embodiment, the informationprocessing device 3 includes a CPU (control unit) 5 and a memory 6, andvarious functions of the information processing device 3 (informationprocesses shown in FIGS. 30 to 33) are implemented as the CPU 5 executesa predetermined information processing program using the memory 6. Notethat the information processing device 3 may have any configuration aslong as the information processes described above can be performed. Inthe present embodiment, an image (panoramic image) is produced by theinformation processing device 3, and the produced image is output to theterminal device 2 and/or the monitor 4, which are display devices. Theinformation processing device 3 may produce a sound to be outputtogether with a panoramic video. The produced sound is output to theterminal device 2 and/or the monitor 4.

The monitor 4 is an example of a display device for displaying theproduced image. The monitor 4 is capable of receiving data transmittedfrom the information processing device 3. Where an image produced in theinformation processing device 3 is transmitted to the monitor 4, themonitor 4 displays the image. The monitor 4 may include a speaker. Wherea sound produced in the information processing device 3 is transmittedto the monitor 4, a speaker outputs the sound.

The controller device 7 is any input device capable of communicatingwith the information processing device 3. The controller device 7 mayinclude an input mechanism such as a button (key) and/or stick, or itmay be a sensor capable of calculating (estimating) the position and/orthe attitude of the controller device 7, such as an acceleration sensorand/or a gyrosensor. The controller device 7 transmits, to theinformation processing device 3, operation data representing anoperation performed on the controller device 7. The informationprocessing device 3 may perform the information processes describedabove based on operation data from the controller device 7. Note that inother embodiments, the information processing system 1 may not includeanother input device (the controller device 7) other than the terminaldevice 2.

[2. Playback of Panoramic Video]

Next, the process of playing a panoramic video to be performed in theinformation processing system 1 will be described. FIG. 2 shows anexample of a panoramic video (image) displayed in the informationprocessing system 1. As shown in FIG. 2, a panoramic image is displayedon the terminal device 2 and the monitor 4 in the present embodiment.Note that in other embodiments, a panoramic image may be displayed on atleast one display device.

Herein, a panoramic image is an image of a wider viewing angle than theviewing angle of the image displayed on a display device. That is,basically, a partial area of a panoramic image is displayed on a displaydevice (note however that where a plurality of display devices are usedas in the present embodiment, the entirety of a panoramic image may bedisplayed on some display devices.). It can also be said that apanoramic image is an image whose viewing direction changes as thedisplay area changes. In the present embodiment, a panoramic image isused which has an omni-direction (360°) viewing angle for the up, down,left and right directions (see FIG. 3). Note however that there may be adead angle in a panoramic image, and the viewing angle may be about180°, for example. While a panoramic image is an actual image(photographed image) captured by a camera in the present embodiment, apart or whole of a panoramic image may be a virtual image (CG image).

In the present embodiment, a panoramic video is played by successivelydisplaying panoramic images on display devices (the terminal device 2and the monitor 4). That is, a panoramic video is formed by a pluralityof panoramic images. The plurality of panoramic images are each assigneda frame number, and a panoramic video is played by displaying panoramicimages in the order of frame number. In the present embodiment, apanoramic video is played in which the viewpoint moves over time. Thatis, as shown in FIG. 2, a panoramic video is played in which a panoramicimage of the surrounding area as one sees from a street is displayedwhile moving along a street (the viewpoint moves along a street).

As shown in FIG. 2, a partial range of a panoramic image is displayed oneach display device (the terminal device 2 and the monitor 4).Hereinafter, a range of a panoramic image to be displayed on a displaydevice is referred to as a “display range”. In FIG. 2, a panoramic imageof a display range such that the viewing direction is the direction inwhich the viewpoint moves (the front direction) is displayed on themonitor 4. In the present embodiment, the display range on the monitor 4is basically fixed to a predetermined range, and may be changed undercertain conditions (the details will be discussed later).

On the other hand, the display range on the terminal device 2 is changedin accordance with the attitude of the terminal device 2. For example,in FIG. 2, the terminal device 2 is in an attitude facing right withrespect to the front direction (the direction from the user (theterminal device 2) toward the monitor 4), and a panoramic image of adisplay range obtained when the viewing direction is facing right withrespect to the moving direction described above is displayed on theterminal device 2. Thus, the information processing system 1 determinesa display range of a panoramic video to be displayed on a display devicebased on an input made on an input device (the terminal device 2). Thus,a user can see a panoramic video while freely changing the viewingdirection (display range) through an operation performed on the terminaldevice 2 (an operation of changing the attitude thereof). Note that inother embodiments, a display range may be determined by any method sothat the display range changes in accordance with an operation by auser.

FIG. 3 shows an example of a method for determining a display area inaccordance with the attitude of the terminal device 2. In the presentembodiment, as shown in FIG. 3, a three-dimensional model 21 is arrangedin a virtual space, and a virtual camera 22 is arranged inside thethree-dimensional model 21. Then, when producing an image of thethree-dimensional model 21 (the inner side surface of thethree-dimensional model 21) as viewed from the virtual camera 22, apanoramic image is rendered on the inner side surface of thethree-dimensional model as a texture, thus producing a panoramic image.Then, the display range is the viewing range of the virtual camera 22 (ahatched area 23 shown in FIG. 3) of the panoramic image. Note that whilethe three-dimensional model 21 is assumed to be cubic in the presentembodiment, the three-dimensional model 21 may have any shape, e.g., aspherical or columnar (e.g., cylindrical) shape.

In FIG. 3, the attitude of the virtual camera 22 for producing an imageto be displayed on the terminal device 2 is controlled in accordancewith the attitude of the terminal device 2. Therefore, the display rangechanges in accordance with the attitude of the terminal device 2. Notethat the information processing device 3 may change the size of adisplay range (zoom in or zoom out), in addition to the position of adisplay range, in accordance with an operation performed on the terminaldevice 2.

On the other hand, a virtual camera (not shown) for producing an imageto be displayed on the monitor 4 is set so that the viewing directionthereof is basically facing in a predetermined reference direction(herein, the moving direction described above. The Z-axis positivedirection in FIG. 3). That is, the display range of the monitor 4 (ahatched area 24 shown in FIG. 3) is set at a position in the referencedirection from the virtual camera. The position of each virtual camerais fixed to a predetermined position in the three-dimensional model 21(the center position of the three-dimensional model 21 in the presentembodiment).

Note that in the present embodiment, the viewing direction of thevirtual camera 22 is controlled so that the relationship between theviewing direction of the monitor 4 (the viewing direction of thepanoramic video displayed on the monitor 4) and the viewing direction ofthe terminal device 2 (the viewing direction of the panoramic videodisplayed on the terminal device 2) generally coincides with therelationship between the orientation of the monitor 4 and theorientation of the terminal device 2. Specifically, when the terminaldevice 2 is in an attitude facing the monitor 4 (this will be referredto as the reference attitude), the viewing direction of the virtualcamera 22 is the same direction as the virtual camera for the monitor 4(that is, the predetermined reference direction). Then, when theattitude of the terminal device 2 changes from the reference attitude,the attitude of the virtual camera 22 changes by an amount that isdetermined in accordance with the amount of change and in the directionin which the attitude has changed. Thus, a user can see a scene in anintended viewing direction by changing the orientation of the terminaldevice 2 while seeing a scene in a certain viewing direction on themonitor 4. By matching the change in the attitude of the terminal device2 with the change in the attitude of the virtual camera 22, theorientation of the terminal device 2 in the real space coincides withthe viewing direction of the virtual camera 22 in the virtual space, andtherefore a user can more realistically feel the space represented bythe panoramic image.

[3. Inputting Comment by User]

In the present embodiment, a user can input a comment (additionalinformation) while a panoramic video is played (see FIGS. 4 and 5). Theinput comment is thereafter output together with a panoramic video whenthe panoramic video is played (see FIGS. 10 to 13). Note that additionalinformation (comment) may be displayed as an image or may be output as asound. Thus, a user can not only simply see a panoramic video, but alsoview a panoramic video together with comments that have been input bythe user or other users. Therefore, according to the present embodiment,the information processing system 1 can present, to a user, moreinformation when displaying a panoramic image. With the presentation ofadditional information, a user can let other users know a range of apanoramic video that the user has seen, or can know a range that otherusers have seen. The details of a method for accepting a comment andoutputting the input comment will now be described.

In the present embodiment, information added to a panoramic video, suchas the comment described above, will be referred to as “additionalinformation”. In the present embodiment, additional information input bya user is a comment (message). Note however that additional informationmay be of any content, and may be, for example, rating information(information representing a rating on an object appearing in a panoramicvideo), user information, or the like, to be described later.

(3-1) Inputting Additional Information

Referring to FIGS. 4 and 5, a method by which a user inputs additionalinformation (comment) will be described. While a panoramic video isplayed, the information processing system 1 accepts an input of acomment on the panoramic video displayed on the terminal device 2. FIG.4 shows an example of an operation for a user to input a comment. In thepresent embodiment, the information processing system 1 accepts an input(touch input) on the screen (on the touch panel 12) where a panoramicvideo is displayed. In FIG. 4, a user can input a comment by handwritingdirectly on the screen where a panoramic video is displayed. Note thatthe specific input method of additional information (comment) may be anymethod, and in other embodiments, the input may be made by using acontrol unit such as a keyboard, a button, or the like, for example, ora sound input may be made if the terminal device 2 includes amicrophone.

For example, an input of additional information may be made by making aninput on a button displayed on a panoramic video. FIG. 5 shows anexample of a panoramic video displaying a button for inputtingadditional information. As shown in FIG. 5, when a rate button 31 isdisplayed on a panoramic video, the information processing system 1 mayaccept an input on the rate button 31 (e.g., an input of touching therate button 31), as an input of additional information. For example, auser may make an input on the rate button 31 a number of times inaccordance with the rating.

When inputting additional information, the information processing system1 may accept an input (position-specifying input) of specifying aposition on a panoramic image displayed on the LCD 11 of the terminaldevice 2. For example, before inputting a comment, a user may be allowedto touch on a position at which the user is inputting a comment. Notethat where additional information is displayed together with a panoramicvideo, additional information may be displayed while being associatedwith the position specified by the position-specifying input, thedetails of which will be described later.

Note that at least while accepting an input of additional information,the information processing system 1 may pause the playback of apanoramic video. For example, the playback of a panoramic video may bepaused from the start to the end of an input of additional information(comment). This is for preventing a user inputting a comment frommissing the content of the video, and for making it easier to input acomment. Note that in other embodiments, the playback of a panoramicvideo may continue even while additional information is being input.

(3-2) Storing Additional Information

When additional information is input, the information processing system1 stores the additional information and information used for outputtingthe additional information. FIG. 6 shows an example of information(referred to as “input information”) to be stored in response to aninput of additional information. In the present embodiment, as shown inFIG. 6, input information including additional information and inputrange information is stored in a storage unit. Although the memory 6 isused as the storage unit in the present embodiment, it may be anystorage device that can be accessed by a computer of the informationprocessing system 1 (the information processing device 3).

Input range information described above is information with which it ispossible to identify the display range at the time when the input ofadditional information was accepted and/or a position within thatdisplay range. Note that information with which it is possible toidentify a display range and/or a position within that display rangewill be hereinafter referred to as “range information”. That is, inputrange information is range information when an input of additionalinformation was accepted. Hereinafter a display range when an input ofadditional information was accepted will be referred to as an “inputrange”, and a position (within a display range) when an input ofadditional information was accepted will be referred to as an “inputposition”.

Input range information may be any information as long as it is possibleto identify at least one of the input range and the input position. Inthe present embodiment, information with which it is possible toidentify the input position is stored as input range information. Forexample, the input position may be a position specified by aposition-specifying input described above, a position at whichadditional information was input (the position at which hand-writtencomment was input, or the position of the rate button 31), or apredetermined position (e.g., a center position) in a display range atthe point in time when additional information was input. Regarding asound input, input range information may be stored which represents adisplay range (or a center position of the display range) at the pointin time when a voice of the user was detected by a microphone, forexample.

As input range information representing an input range and/or an inputposition, information of a vector representing the attitude (viewingdirection) of the virtual camera 22 is stored in the present embodiment(see FIG. 6). For example, an input position can be represented by avector representing a direction from the position of the virtual camera22 toward the input position on the three-dimensional model 21. An inputrange can be represented by a vector representing a direction from theposition of the virtual camera 22 toward the center position of theinput range on the three-dimensional model 21, and a vector representingthe attitude with respect to the rotation about that direction. Notethat in other embodiments, input range information may includeinformation (zoom information) indicating the size of a display range,in addition to information of a vector. In other embodiments,information representing a range or a position on the three-dimensionalmodel 21 may be stored as input range information.

In the present embodiment, additional information is informationrepresenting the content of a comment (see FIG. 6). Specifically, textinformation representing the content of a comment may be stored asadditional information, or when a handwriting input on the touch panel12 is accepted, information representing the trace of the input touchposition may be stored as additional information. The content of theinformation stored as additional information is not limited to thecontent of a comment.

As shown in FIG. 6, input information further includes time informationin the present embodiment. Time information is information with which itis possible to identify a playback time point at which an input ofadditional information was accepted. Note that a playback time point isa point along the time axis of a panoramic video. Hereinafter, aplayback time point at which an input of additional information wasaccepted will be referred to as an “input time point”. In the presentembodiment, time information is information of a frame number (denotedas the “i^(th) frame” in FIG. 6) of a panoramic image that was displayedat the input time point. Time information may be any information withwhich it is possible to identify a playback time point along the timeaxis of a panoramic video. For example, time information may beinformation representing the elapsed time since the start of theplayback of a panoramic video. Where the viewpoint of a panoramic videomoves through a predetermined space over time as in the presentembodiment, information representing the position of the viewpoint inthe predetermined space may be stored as time information. Note that inother embodiments, input information does not need to include timeinformation.

As shown in FIG. 6, in the present embodiment, input information furtherincludes user information. User information is information with which itis possible to identify a user who has input the additional information.User information may be of any specific content. In the presentembodiment, before playing a panoramic video, user identificationinformation (e.g., the name (nickname) of a user) is registered, and theregistered identification information is stored as user information.Note that in other embodiments, input information does not need toinclude user information.

(3-3) Outputting Additional Information

When a panoramic video is played, with input information includingadditional information stored as described above, the informationprocessing system 1 outputs additional information together with thepanoramic video. In the present embodiment, the information processingsystem 1 controls additional information to be output together with thepanoramic video based on input range information. Typically, while apanoramic video is played, if the input time point has been reached andif the input position is included in the current display range,additional information is displayed together with the panoramic video(see FIG. 10, etc.). Note that additional information is not displayedif, for example, the input time point has not been reached or if theinput position is not included in the current display range (if thecurrent viewing direction is facing in a different direction from thatwhen the additional information was input). How additional informationis output will now be described.

(3-3-1) Method for Determining Output of Additional Information

In the present embodiment, the information processing system 1determines whether or not to output additional information using timeinformation and input range information. Specifically, the informationprocessing device 3 determines (a) whether the display range of thepanoramic video being played satisfies a condition regarding a displayrange and/or a position identified by input range information, and (b)whether the timing of the panoramic video being played satisfies acondition regarding a playback time point identified by timeinformation.

FIG. 7 shows an example of a method for making a decision on a conditionregarding input range information. As the decision regarding (a) above,the information processing device 3 may determine whether the currentdisplay range overlaps a comment input position (a position identifiedby input range information) (see FIG. 7). For example, where the inputposition is included in the current display range as shown in FIG. 7, itis determined that the condition (a) is satisfied. Thus, it is possibleto determine whether the comment input position is being displayed onthe terminal device 2.

FIG. 8 shows another example of a method for making a decision on acondition regarding the input range information. In other embodiments,as the decision regarding (a) above, the information processing device 3may determine whether the current display range overlaps the displayrange at the time of comment input (the display range identified by timeinformation). For example, where the input range at least partiallyoverlaps the current display range as shown in FIG. 8, it is determinedthat the condition (a) is satisfied. Thus, it is possible to determinewhether the input range at least partially overlaps the current displayrange. That is, it is possible to determine whether at least a portionof the display range at the time of comment input is currentlydisplayed.

Note that the decision described above using FIGS. 7 and 8 may be madeby comparing a position (or a range) on the three-dimensional model 21,or by comparing the vector representing the attitude of the virtualcamera 22.

As the decision regarding (b) above, it may be determined whether thecurrent playback time point of the panoramic video is within apredetermined output period including the playback time point (inputtime point) identified by time information. FIG. 9 shows an example of acomment output period. As shown in FIG. 9, the output period may be setas a period that starts a predetermined amount of time before thecomment input time point (the playback time point indicated by timeinformation) and ends upon passage of a predetermined output time, forexample. Note that the output period may be set based on the commentinput time point, or may be set by the user who has input the comment.Where the comment is text information, the output period may be set to alength determined in accordance with the number of characters (e.g., sothat it is longer for a larger number of characters).

If the decisions (a) and (b) are both affirmative, the informationprocessing system 1 determines to output additional information togetherwith a panoramic video. In the present embodiment, a comment (additionalinformation) is output when the current display range includes the inputposition therein during an output period that includes the comment inputtime point therein.

(3-3-2) Manner of Outputting Additional Information

FIG. 10 shows an example of an image in which additional information isdisplayed together with a panoramic video. When the informationprocessing system 1 determines to output additional information, acomment image 32 is displayed on a panoramic video on the terminaldevice 2 as shown in FIG. 10. Thus, in the present embodiment, if acomment is input for an object (herein, a building) appearing in apanoramic video, the panoramic video is displayed with the commentattached to the object when the panoramic video is played thereafter.

Now, the comment image 32 indicates an input position on a panoramicimage as shown in FIG. 10. That is, if the input position is included inthe current display range, the information processing system 1 displaysan image of the additional information in association with the inputposition. Thus, it is possible to allow a user to clearly recognize theposition at which the comment has been made. For example, in FIG. 10, auser can easily grasp the building which has been commented on. Notethat FIG. 11 shows another example of additional information to beoutput. As shown in FIG. 11, the comment image 32 may be displayed atthe input position. For example, where a comment has been made by anhandwriting input on the touch panel 12, the hand-written characters maybe displayed (as they are) at the input position. Note that while animage of additional information is displayed while being associated withthe input position, it does not need to be displayed at the inputposition. For example, when the comment image 32 is displayed at theinput position, it may possibly make it more difficult to see the objectwhich has been commented on. Therefore, the comment image 32 may bedisplayed at a position slightly off the input position (as shown inFIG. 10). Note that the term “image” as used in the presentspecification includes those representing characters (text information).

As shown in FIG. 10, in the present embodiment, a user image 33representing the name (Mr. A) of the user who has input the comment isdisplayed together with the comment image 32. Thus, the informationprocessing system 1 outputs additional information, and user informationassociated with additional information. Note that user information to beoutput may be the name of a user, a character (e.g., a characterresembling a user) set for each user, an image of a likeness of a user,or the like. According to the present embodiment, a user who seesadditional information can not only know the content of additionalinformation, but also know where in a panoramic video another user whohas input the additional information was seeing. Now, a panoramic videois a video whose area cannot be contained within one screen, and adisplay range on the terminal device 2 can be freely controlled by theuser. Therefore, the position of a display range and the transitionthereof while viewing a panoramic video differs from one user toanother. Therefore, it is an interesting piece of information for a userto know where in a panoramic video another user was seeing, as in thepresent embodiment. Note that user information may be displayed togetherwith additional information also in other manners of display (FIGS. 11to 13, and the like) other than the manner of displaying additionalinformation shown in FIG. 10.

Note that in other embodiments, user information may be displayed asadditional information itself. For example, only an image representinguser information (e.g., an image of a likeness of a user) may bedisplayed at an input position. Thus, a user who has input additionalinformation can let another user, who sees the additional information(user information), know that the user was looking at a display rangeincluding the input position therein.

Note that where the viewpoint of a panoramic video moves over time as inthe present embodiment, each object appearing in the panoramic videomoves in the opposite direction to the viewpoint moving direction on thepanoramic image. Therefore, in such a case, the information processingsystem 1 may display additional information while moving additionalinformation following the movement of an object associated with theadditional information. Note that an “object associated with additionalinformation” is an object that is displayed at the input position at theinput time point, in other words, an object arranged at a position thatis identified by the input range information at the input time point.Specifically, additional information may be displayed so as to move overtime in the opposite direction to the viewpoint moving direction. Forexample, where the viewpoint moving direction is the forward direction,additional information may be displayed so as to move in thefront-to-back direction. Note that where the moving direction and themoving velocity of the viewpoint are known, it is possible to identifythe direction and the velocity of the movement of an object (a building,or the like) that is fixedly arranged in the space on a panoramic video,and it is possible to calculate the algorithm for moving additionalinformation based on the moving direction and the moving velocity of theviewpoint. Therefore, by moving additional information according to thisalgorithm, it is possible to display additional information while movingthe additional information following the movement of an object. Notethat additional information does not need to exactly match the positionof the object, but may be displayed somewhat in the vicinity of theobject.

As a method for moving additional information following the movement ofan object, object information representing the position and/or the rangeof the object (on the panoramic image) may be prepared together with thepanoramic image. FIG. 12 shows an example of object information. Asshown in FIG. 12, where an object (a building in FIG. 12) appears in apanoramic image, object information representing the range of thisobject (e.g., a dotted-line area 34 shown in FIG. 12) is stored togetherwith a panoramic video. Note that for each of a plurality of panoramicimages of a panoramic video, the information processing system 1 storesobject information of one or more objects appearing therein while theobject information is associated with the panoramic image. Whenaccepting an input of a comment, the information processing device 3determines whether the comment input position is within the range of theobject using the object information. If the input position is within therange of the object, the information processing device 3 storesinformation representing the object as input range information. Whenplaying a panoramic video, the information processing device 3 displaysan image of a comment while the image is associated with the position ofthe object at least on the condition that the object is included in thedisplay range of the panoramic video being played. Specifically, thecomment is displayed while being associated with the position (or range)of the object identified by the object information. Therefore, it ispossible to display additional information at an accurate positiondetermined in accordance with the position of an object. With the methodof using object information described above, even with an object thatseems to be moving on a panoramic image by moving through the space(irrespective of the movement of the viewpoint), additional informationcan be displayed at a position determined in accordance with the object.

Note that where an image of a map to be described later (see FIG. 25) isprovided and a position on the map can be associated with a position ona panoramic video, object information may be stored while beingassociated with the map (as information representing the position orrange on the map). Then, the information processing device 3 mayidentify a range on the map corresponding to a display range, andidentify the position or range of an object on a panoramic video fromobject information that is associated with the identified range.

Where object information is stored as input range information,additional information can be displayed during a period in which anobject represented by the object information appears in a panoramicvideo. That is, the output period of additional information can becontrolled (without using the time information) by storing objectinformation as input range information.

In other embodiments, additional information may be displayed at apredetermined position on the screen of the terminal device 2. FIG. 13shows another example of an image in which additional information isdisplayed together with a panoramic video. As shown in FIG. 13, thecomment image 32 may be displayed in a predetermined area of the screen(a lower area in FIG. 13) (irrespective of the position of the objectassociated with the comment). For example, where the comment inputposition is not stored as input range information, it is difficult todisplay the comment at a position associated with the object, andtherefore the comment may be displayed at a predetermined position onthe screen. That is, where the display range of the panoramic videobeing played at least partially overlaps the display range whenadditional information was accepted, the information processing system 1may display the additional information at a predetermined position onthe screen of a display device. Thus, it is possible to easily identifythe position at which additional information is to be displayed.Moreover, as shown in FIG. 13, the additional information may bedisplayed in an area different from the area where a panoramic video isdisplayed. Thus, a panoramic video can be displayed in an easy-to-viewmanner since the panoramic video is not hidden by additionalinformation.

(3-3-3) Output Content of Additional Information

In other embodiments, the content to be output as additional informationis not limited to a comment input by a user, but may be otherinformation based on a user input. For example, where rating informationis input by using the rate button 31 described above, rating informationrepresenting the input count on the rate button 31 may be stored asadditional information, and an image representing the input count may beoutput as additional information. FIG. 14 shows another example of animage in which additional information is displayed together with apanoramic video. In such a case, a bar 35 whose length is associatedwith the input count may be displayed as additional information as shownin FIG. 14, for example. Note that the bar 35 is displayed at a positionbased on (e.g., a position in the vicinity of) the input position ofadditional information (the position of the rate button 31).

Note that the information processing system 1 may not only outputinformation based on one piece of additional information, but may alsooutput information based on a plurality of pieces of additionalinformation. For example, in the example shown in FIG. 14, the length ofthe bar 35 may be set to represent the sum of the input counts from aplurality of pieces of input information (i.e., the total count ofinputs made on the rate button 31 by a plurality of users), or theaverage value thereof. Thus, where additional information is informationrepresenting a numerical value (herein, the input count), theinformation processing system 1 may output information representing anumerical value (herein, the sum of input counts) obtained based onnumerical values represented by a plurality of pieces of additionalinformation to be output. Note that the “plurality of pieces ofadditional information to be output” may be regarded as such on acondition that they are pieces of additional information of the samekind whose output content should be grouped together (e.g., pieces ofadditional information regarding the rate button 31). Thus, it ispossible to present, to a user, statistical information based on aplurality of users (or a plurality of views). The information processingsystem 1 may output information representing the number of pieces ofadditional information, of the plurality of pieces of additionalinformation, that satisfy a predetermined condition. For example,information representing the number of pieces of additional informationof the same kind may be output.

In other embodiments, the information processing system 1 may perform acontrol so that the manner of outputting additional information changesin accordance with the change in the display range of the terminaldevice 2. FIG. 15 shows another example of an image in which additionalinformation is displayed together with a panoramic video. Note that inFIG. 15, as in the case shown in FIG. 14, it is assumed that the inputcount on the rate button 31 is stored as additional information. Then,the information processing system 1 displays a bar 36 whose length isassociated with the input count and whose length is associated with thepositional relationship between the current display range and the inputrange. That is, the bar 36 is displayed to be longer as the input countis higher. Moreover, the bar 36 is displayed to be longer as the ratebutton 31 (the object associated with the rate button 31) is closer tothe center of the display range. Note that the information processingsystem 1 can determine the positional relationship based on input rangeinformation. The bar 36 is always displayed on the screen of theterminal device 2, irrespective of the current display range. Therefore,the length of the bar 36 changes in accordance with the movement of thedisplay range. Note that the length of the bar 36 may be determinedwhile taking into consideration the input count on another rate buttonin addition to the rate button 31. That is, the bar 36 may be displayedto be longer as there exist more objects with higher ratings (higherinput counts) within the display range. Thus, the user can look forobjects with higher ratings by moving the display range while looking atthe bar 36.

As described above, the information processing system 1 may determine,based on input range information, the positional relationship betweenthe display range of the panoramic video being played and the displayrange when additional information was accepted, and change the manner ofoutputting additional information (the length of the bar) based on thepositional relationship. Thus, information that changes in accordancewith the display range can be presented to the user, and the user cancontrol the display range based on the information.

Note that in other embodiments, the information processing system 1 mayoutput a sound representing additional information, in addition to orinstead of displaying an image representing additional information. Notethat the sound may be output from the terminal device 2, from a speakerof the monitor 4, or from both.

(3-3-4) Output Destination of Additional Information

In the present embodiment, additional information is displayed on theterminal device 2. The terminal device 2 is a display device whosedisplay range is controlled by a user. Therefore, since a user canrecognize additional information when moving the display range bycontrolling the terminal device 2, the user is allowed to more certainlyrecognize additional information.

Note that in other embodiments, additional information may be displayedon a display device (the monitor 4) different from the terminal device2. For example, the information processing system 1 may displayadditional information on the monitor 4 while displaying, on theterminal device 2, a panoramic video of a display range determined basedon an input made on the terminal device 2. Thus, a panoramic videodisplayed on the terminal device 2 can be presented in an easy-to-viewmanner. Where there is another user seeing the monitor 4 other than theuser using the terminal device 2, the additional information can bepresented also to the other user. Additional information may bedisplayed on the monitor 4, when the input position of the additionalinformation is included within the display range of the monitor 4, orthe display range on the monitor 4 at least partially overlaps the inputrange of the additional information. Thus, it is possible to increasethe chance for a user to see additional information.

In other embodiments, the information processing system 1 may display anentire panoramic image on the monitor 4, or may display an image of thespace represented by the panoramic image (e.g., a bird's-eye view or amap) on the monitor 4. Then, additional information may be displayedtogether with (while being superimposed on) the image displayed on themonitor 4.

(3-4) Additional Information from Other Users

Other than storing input information by an input made on the terminaldevice 2, the information processing system 1 may obtain inputinformation from another device outside the information processingsystem 1, and output additional information included in the inputinformation together with a panoramic video. For example, theinformation processing device 3 may be capable of communicating with anexternal device via a network such as the Internet, and may obtain inputinformation from the external device. Note that the “external device”may be an information processing device included in a system having thesame function as the information processing system 1, for example, ormay be a server storing input information. Thus, a user can see apanoramic video with additional information (comment) added thereto,which has been input by another user (e.g., a friend or a celebrity). Byseeing additional information from another user, a user can know wherein a panoramic video the other user was seeing.

[4. Outputting Guide]

In the present embodiment, guide information is output, in addition toadditional information (comment), while a panoramic video is played.Guide information is information for guiding a target on a panoramicimage. Herein, a target is a position and/or a range set on a panoramicimage. For example, the position at which additional informationdescribed above is displayed may be set as a target, a positionspecified by another user may be set as a target, or a display rangeobtained from history information to be described later may be set as atarget. Guide information can also be said to be information fornotifying the user of a location on a panoramic image where a target isset.

FIG. 16 shows an example of a positional relationship between a displayrange and a target. In FIG. 16, the display range is an area 23, and thetarget is outside the area 23. Therefore, in a state shown in FIG. 16,information associated with the position of the target is not output.Then, even though the position of the target is very close to thedisplay range, a user may possibly not notice the presence of the targetand therefore not move the display range toward the target.

In view of this, in the present embodiment, the information processingsystem 1 outputs guide information together with a partial area of apanoramic video in order to let the user know of the target. FIG. 17shows an example of guide information. Note that the guide informationshown in FIG. 17 is what is displayed when the relationship between thedisplay range and the target is as shown in FIG. 16. As shown in FIG.17, in the present embodiment, a guide image 41 representing an arrowindicating the right direction ward is displayed as the guideinformation. Thus, it is possible to notify a user of the direction inwhich the user should move the display range in order for the positionor range of the target to be included within the display range. Thus,where guide information is output together with a panoramic video, it ispossible to guide a user to move the display range toward the target bypresenting guide information to the user even if the current displayrange does not include the target. Note that guide information may bedisplayed as an image or output as a sound.

(4-1) Setting Target

The information processing device 3 sets a range and/or position on apanoramic image as the target. The position and/or range set as thetarget may be any position and/or range. For example, the target may bea position and/or a range where some information is presented to theuser. Specifically, where additional information is displayed on apanoramic video, the information processing device 3 may set theposition (and/or range) associated with additional information as thetarget. Then, with the guide information, it is possible to guide a userto move the display range so that additional information is displayed inthe display range. Note that the image shown in FIG. 17 is an example ofan image to be displayed when the position of a comment, which is anexample of additional information, is set as the target. History-relatedinformation produced from history information is information that canpossibly be additional information, and the position associated withhistory-related information which is additional information may be setas the target. Note that history information represents an operation bya user while a panoramic video is played and/or information determinedby such an operation, details of which will be described later.

The information processing device 3 may set a position (or range)specified by another user different from the user of the terminal device2 as the target. That is, the information processing device 3 may set,as the target, a range or a position specified by an input made onanother input device (the controller device 7) different from the inputdevice (the terminal device 2) used for controlling the display range.For example, where a panoramic video is displayed on another displaydevice (the monitor 4) different from the display device (the LCD 11) ofthe terminal device 2, the information processing device 3 may set aposition (and/or range) on a panoramic video displayed on the otherdisplay device based on an input made on the other input device. Then,the specified position (and/or range) is set as the target.

FIG. 18 shows an example of an image displayed on each display devicewhere a target is set on another display device. In FIG. 18, a cursor 43is displayed together with a panoramic video on the monitor 4. Thecursor 43 is controlled by the controller device 7. Another user usingthe controller device 7 moves the cursor 43 to an intended position, andperforms a predetermined operation (e.g., an operation of pressing abutton of the controller device 7). The information processing device 3sets, as the target, a position on a panoramic image at the point intime when the predetermined operation was performed. Note that while theoperation of setting the target using the cursor 43 is described as anexample in the present embodiment, the method of operation for settingthe target may be any method. While an image (image representing“Attention!”) 44 representing additional information is displayed at theposition of the target in FIG. 18, such an image does not need to bedisplayed. When the target is set, the information processing device 3outputs (displays) guide information associated with the target on theterminal device 2. For example, in FIG. 18, since the target is set tothe left with respect to the display range on the terminal device 2, aguide image 45 representing an arrow indicating the left direction isdisplayed as guide information on the terminal device 2.

Where the target can be set by an input made on the other input deviceas described above, another user different from the user using theterminal device 2 can set the target. For example, if another userinputs a “recommended location” as the target using another inputdevice, it is possible to let the user of the terminal device 2 know ofthe “recommended location”.

Where the position (and/or range) on a panoramic video displayed on theother display device can be specified as the target, guide informationfor the target specified on the screen of one display device can beoutput on another display device. Therefore, a user who sets a targetcan easily set a target at a location outside the display range which isbeing seen by another user. Note that in other embodiments, theinformation processing device 3 may set a position (and/or range) to bethe target based on an input made on the terminal device 2, and displayguide information associated with the set target on the monitor 4.Effects similar to those described above can be achieved also in thismanner.

An object to be set as the target is not limited to a position on apanoramic image, but may also be a range. For example, the informationprocessing device 3 may set, as the target, a display range used in aseparate instance of playback of a panoramic video. Herein, “a separateinstance of playback of a panoramic video” refers to a panoramic videobeing played in a separate instance from that in which the display rangeis determined in accordance with an operation by a user (playback on theterminal device 2 in the present embodiment). For example, if apanoramic video has been played in the past, that can be referred to as“a separate instance of playback of the panoramic video”. When apanoramic video is played on another display device (e.g., the monitor4) different from the display device (e.g., the terminal device 2) onwhich the display range is determined in accordance with an operation bya user, that can be referred to as a separate instance of playback ofthe panoramic video”. As described above, a display range used in aseparate instance of playback of the panoramic video may be set as thetarget, and guide information associated with the display range may beoutput.

FIG. 19 shows an example of an image displayed on each display devicewhere a display range is set as the target. In FIG. 19, a panoramicvideo of a display range such that the viewing direction of the virtualcamera 22 is facing in the front direction is displayed on the monitor4, and a panoramic video of a display range such that the viewingdirection is facing right with respect to the front direction isdisplayed on the terminal device 2. Therefore, on the terminal device 2,a guide image 47 representing an arrow indicating the left direction,which is the direction toward the display range of the monitor 4 withrespect to the display range on the terminal device 2, is displayed asguide information. On the monitor 4, a guide image 46 representing anarrow indicating the right direction, which is the direction toward thedisplay range of the terminal device 2 with respect to the display rangeon the monitor 4, is displayed as guide information. Therefore, when apanoramic video is displayed on another display device different from adisplay device, the information processing device 3 may set, as thetarget, the display range displayed on the other display device. Notethat while each of the two display devices is outputting guideinformation associated with the display range on the other displaydevice in FIG. 19, the guide information may be output only one of thedisplay devices.

The information processing device 3 may set, as the target, a displayrange used when a panoramic video was played in the past. For example,the display range used when a panoramic video was played in the past canbe identified based on history information to be described later. Theinformation processing device 3 may set, as the target, the displayrange identified based on the history information. That is, theinformation processing device 3 may output, together with a panoramicvideo, guide information representing a direction facing the displayrange when the panoramic video was played in the past, with respect tothe current display range (see FIG. 21). Note that history-relatedinformation to be discussed below in “(history-related informationrepresenting the direction of the display range)” is one of guideinformation.

As described above, where a display range in a separate instance ofplayback of a panoramic video is set as the target, a user can benotified of a display range that another user was seeing (or is seeing),and the user can be guided to directing the display range of the usertoward the display range of the other user.

(4-2) Storing Target Information

When the target is set, the information processing device 3 storestarget information representing the set target. In the presentembodiment, target information includes target position information andtarget period information. Target position information is informationwith which it is possible to identify the position and/or range set asthe target. The position and/or range set as the target can berepresented in a similar manner to the range information (input rangeinformation) described above, and it may be represented by informationof a vector representing the attitude (viewing direction) of the virtualcamera 22 or as a range and/or position on the three-dimensional model21, for example.

Target period information is information with which it is possible toidentify the period in which the target is set. For example, the targetperiod information may represent the start point and the end point ofthat period, or may represent the point in time when the target was set(if the period can be set based on the point in time). The period inwhich the target is set can be represented in a similar manner to thetime information described above, and it may be represented by the framenumber of the panoramic image, for example.

Note that where the target to be set is associated with the additionalinformation described above (where the target is set at the position ofadditional information), the information processing device 3 may set thetarget information based on input information associated with additionalinformation. That is, target position information may be set based oninput range information included in input information. Target periodinformation may be set based on time information included in inputinformation. Moreover, input range information may be used as it is astarget position information, and the time information may be used as itis as the target period information. Then, the information processingdevice 3 does not need to additionally store target informationseparately from the input information (by storing input information,target information can be assumed stored).

(4-3) Method for Determining Output of Guide Information

In the present embodiment, the timing for outputting guide informationis determined based on the period in which the target associated withthe guide information is set. While the specific method for determiningthe timing may be any method, the information processing system 1displays guide information during the period described above in thepresent embodiment. Note that guide information may be displayed,starting a predetermined amount of time before the period and ending atthe end of the period.

In the present embodiment, where the target is located within thedisplay range, guide information associated with the target is notoutput. For example, where additional information is displayed withinthe current display range, guide information associated with theadditional information is not displayed. Therefore, it is possible todisplay a panoramic video and additional information in an easy-to-viewmanner without displaying unnecessary guide information.

Where a plurality of targets are set in a panoramic video, theinformation processing system 1 may output guide information for eachtarget that satisfies a predetermined condition. The predeterminedcondition may be, for example, the target being one that is set by aparticular user (e.g., one associated with additional information inputby the particular user), the target being one of a particular type(e.g., one associated with additional information of a comment), thetarget being one that is set within a predetermined range with respectto the current display range, or the like. Thus, since guide informationis output only for targets that satisfy the condition, it is possible toreduce the number of pieces of guide information to be output, and todisplay a panoramic video in an easy-to-view manner. Note that thepredetermined condition may be specified by the user, for example.

In the present embodiment, the guide information represents thedirection toward the target (the range and/or position set for thetarget) with respect to the current display range (see FIG. 17). Theinformation processing device 3 determines the direction based on thecurrent display range and the range and/or position set for the target.For example, the direction can be calculated by using a vectorrepresenting the viewing direction of the virtual camera 22. Morespecifically, the direction can be calculated as a direction toward avector representing the position set for the target (or the centerposition of the range set for the target) with respect to a vectorrepresenting the center position of the current display range.

(4-4) Manner of Outputting Guide Information

Guide information is an image and/or a sound representing therelationship between the display range and the target. In the presentembodiment, guide information is output, which represents the directiontoward the target with respect to the display range, as described above.While guide information is the guide image 41 representing an arrow asshown in FIG. 17 in the present embodiment, guide information may beoutput in any manner of output such that it is possible to notify theuser of the direction. For example, in other embodiments, theinformation processing system 1 may output a sound representing thedirection together with an image representing the direction (or insteadof the image). Note that the sound is output from the terminal device 2and/or the monitor 4. The user may be made to guess the direction towardthe target with respect to the display range, using an image of a map tobe described later (see FIG. 25). With the guide image of the presentapplication, it is possible to present the direction of the target withrespect to the display range in an intuitive and easy-to-understandmanner.

In the present embodiment, the information processing device 3 mayoutput user information together with guide information. Userinformation to be output represents the user who has input additionalinformation associated with guide information. For example, in FIG. 17,a user image 42 representing the name (Mr. A) of the user who has inputa comment is displayed together with the guide image 41. Thus, a usercan decide whether or not to see the comment (i.e., whether or not tomove the display range toward that comment) based on who is the user whohas input the comment.

The information processing device 3 may output information representingthe type of the target, together with the guide information. Forexample, in FIG. 17, an image representing the type of the target (animage representing the “comment of Mr. A”) 42 is displayed together withthe guide image 41. The type of the target is, for example, the type ofinformation to be output when the position and/or range of the target isincluded within the display range. Specifically, informationrepresenting the type of the target may be the type regarding thecontent to be output, e.g., “comment” or “rating information”, or it maybe the name of the person who has input the information, e.g., “Mr. A”or “Mr. B”. Thus, it is possible to notify the user of the informationwhich has been set as the target. The user can know the type of theinformation to be output at the location of the target, before actuallymoving the display range to the target.

Note that where the position of additional information moves on apanoramic image, the target associated with additional information isalso moved in accordance with the movement of additional information.Then, the information processing system 1 changes the directionrepresented by guide information in accordance with the movement of thetarget (additional information). That is, while guide information isoutput, the information processing system 1 may repeatedly calculate thepositional relationship between the display range and the target at thatpoint in time, and repeatedly calculate the direction represented byguide information based on the positional relationship. Thus, even whenthe target moves, it is possible to output guide informationrepresenting an accurate direction following the movement.

(4-5) Output Destination of Guide Information

In the present embodiment, guide information is displayed on the displaydevice (the terminal device 2) where additional information is displayed(see FIGS. 17 and 18). Therefore, when moving the display range bycontrolling the terminal device 2, the user can recognize guideinformation, and therefore it is possible to allow the user to morecertainly recognize guide information. Note that in other embodiments,guide information may be displayed on a different display device (themonitor 4) from the display device where additional information isdisplayed. Thus, a panoramic video displayed on the terminal device 2can be presented in an easy-to-view manner. In other embodiments, guideinformation may be displayed on both of the two display devices.

(4-6) Content of Additional Information Associated with GuideInformation

Where a target is set associated with additional information, thecontent of additional information associated with guide information isnot limited to the additional information which has been input by auser. Additional information associated with guide information may beany information that is output together with a panoramic video when acondition regarding the display range and the playback time point duringthe playback of the panoramic video is satisfied. That is, additionalinformation may be any information that is output together with apanoramic video when the panoramic video has reached a predeterminedplayback time point and the display range includes a predeterminedrange. For example, additional information may be information that isobtained from an external device as described in (3-4) above, or may beinformation that is pre-registered in a panoramic video. Thehistory-related information to be described later is a type ofadditional information, and guide information may be set for thehistory-related information.

[5. Output Reflecting Another Instance of Playback of Panoramic Video]

In the present embodiment, the information processing system 1 canoutput a panoramic video while reflecting the playback content from “aseparate instance of playback of the panoramic video” described above.Note that the playback content is a concept including a display range ofa panoramic video, and a position in the display range (e.g., a positionspecified by a user). For example, the information processing system 1can output, together with a panoramic video, information based on ahistory obtained when a panoramic video was played in the past. Whereone panoramic video is played simultaneously on a plurality of displaydevices, for example, the information processing system 1 can outputinformation representing the display range being played on anotherdisplay device to a different display device together with a panoramicvideo. As described above, in the present embodiment, the informationprocessing system 1 outputs the current instance of playback of thepanoramic video while reflecting the playback content from a separateinstance of playback of the panoramic video in the past or on anotherdevice. Thus, it is possible to present, to a user, more informationwhen displaying a panoramic image. For example, it is possible topresent, to a user, interesting pieces of information such as where in apanoramic video another user was seeing.

Output modes of outputting a panoramic video while reflecting theplayback content from a separate instance of playback of a panoramicvideo include the first output mode in which the playback content from apast instance of playback is reflected, and the second output mode inwhich the playback content from a concurrent (real-time) instance ofplayback on another device is reflected. The first output mode is anoutput mode in which the playback content from a past instance ofplayback of a panoramic video is reflected in a later instance ofplayback of the panoramic video. The second output mode is an outputmode in which where the same panoramic video (different display rangesmay be used) is played on at least two display devices, the playbackcontent from one display device is reflected in the playback of thepanoramic video on the other display device. The first output mode andthe second output mode will now be described.

(5-1) First Output Mode

In the first output mode, the information processing system 1 stores, ashistory information, information regarding the display range duringplayback of a panoramic video, and outputs based on the stored historyinformation when the panoramic video is played thereafter (second andthird playback modes to be described later). That is, the informationprocessing device 3 obtains the range information described above whilea panoramic video is played, and stores the range information, ashistory information, in a storage unit that can be accessed by theinformation processing device 3. Then, after the history information isstored, the information processing device 3 outputs information based onthe history information.

As described above, in the present embodiment, the display rangecontrolled by a user while a panoramic video is played can be stored ashistory information. Thus, it is possible to output in accordance withthe history information while playing a panoramic video. For example,the information processing device 3 can produce additional informationbased on history information to output additional information togetherwith a panoramic video, or play a panoramic video controlling thedisplay range based on a history. Thus, a user can know which places inthe panoramic video are much seen, and/or which locations in thepanoramic video have been seen by another user, for example. That is,according to the present embodiment, it is possible not only to simplypresent a panoramic video, but also to present information interestingto a user together with a panoramic video. How a history is stored andhow a panoramic video is played based on the history will now bedescribed in detail.

(5-1-1) Storing History Information

The information processing system 1 stores history information describedabove while a panoramic video is played. In the present embodiment, theinformation processing system 1 stores history information, in whichtime information with which it is possible to identify a playback timepoint in a panoramic video is associated with range information withwhich it is possible to identify a display range at the playback timepoint or a position within the display range. History information isstored in the storage unit described above (herein, the memory 6). FIG.20 shows an example of history information. As shown in FIG. 20, historyinformation includes a plurality of pairs of the time information andthe range information. In the present embodiment, time information isinformation representing a frame number of a panoramic image, and rangeinformation is information of a vector representing the attitude of thevirtual camera 22. While a pair of time information and rangeinformation is stored for every frame of a panoramic video in thepresent embodiment, such pairs in other embodiments may be stored atpredetermined time intervals or only for key frames, or such a pair maybe stored at a point in time when a predetermined condition issatisfied. By storing history information in which time information andrange information are associated together as described above, it ispossible to identify a display range at each playback time point in apanoramic video.

As shown in FIG. 20, in the present embodiment, user informationrepresenting a user who is associated with history information is storedwhile being associated with history information. Note that “a userassociated with history information” is the user who has performed anoperation for a display range that is represented by the historyinformation. While user information may be any information with which itis possible to identify a user, the pre-registered name of the user isstored as user information in the present embodiment.

Note that in other embodiments, history information may includeinformation other than those described above. For example, historyinformation may include information representing the size of a displayrange (zoom information). History information may also includeinformation (referred to as playback control information) representing aplayback-related operation performed by a user while a panoramic videois played. For example, playback control information is informationrepresenting an operation such as pause, rewind, fast-forward, etc., ofthe playback of a panoramic video. Playback control information mayfurther include a playback time point at which such an operation hasbeen performed. In some cases, a panoramic video is of a multi-cameratype. With a multi-camera type, a plurality of panoramic videos of asingle space as viewed from different viewpoints are prepared, and theviewpoint can be switched from one to another while playing thepanoramic video. In such a case, information of the viewpoint may bestored while being included in the history information. Where a comment(additional information) described above is input by a user, the inputinformation and the history information described above may be storedwhile being associated with each other. That is, the input informationmay be stored as a part of the history information.

(5-1-2) Playing Panoramic Video Using History Information

The information processing system 1 can play a panoramic video based onthe history information. That is, the information processing system 1outputs a panoramic video while changing the output content (displaymode) thereof in accordance with the history information. Thus, it ispossible not only to simply present a panoramic video, but also topresent history information (or information based on historyinformation) to a user. Now, methods for outputting a panoramic videowhile changing the output content thereof in accordance with the historyinformation may include (A) to (C) below, for example.

(A) Method of Outputting History-Related Information

In the present embodiment, the information processing system 1 canoutput, together with a panoramic video, information produced based onhistory information. Hereinafter, information produced based on historyinformation will be referred to as “history-related information”. Theinformation processing system 1 produces various history-relatedinformation to be described below from history information, and outputsthe history-related information when a panoramic video is played. Thus,a user can obtain information regarding the history of another user byseeing the history-related information while a panoramic video isplayed.

(History-Related Information Representing Direction of Display Range)

FIG. 21 shows an example of an image of history-related information. Thehistory-related information shown in FIG. 21 presents, to a user, thedirection of a display range identified by the history information. Thatis, in FIG. 21, an image 51 representing the direction of the displayrange identified by the history information (range information includedin history information) with respect to the current display range isdisplayed as history-related information. Note that the image 51, whichis history-related information, can also be said to be guide informationdescribed above.

Specifically, the information processing system 1 identifies a range orposition on a panoramic image based on one of the range informationincluded in the history information that is associated with the timeinformation corresponding to the current playback time point. Then, thedirection of the calculated range or position is calculated with respectto the current display range. Note that this direction can be calculatedby a method similar to that for a direction represented by guideinformation described above. The information processing system 1displays the image 51 representing the calculated direction on theterminal device 2. Note that the image 51 is displayed at a positiondetermined in accordance with the calculated direction on the screen ofthe LCD 11 (at a position on the left side of the screen in FIG. 21since the calculated direction is left).

As the image 51 is presented, a user can know where in a panoramic videoanother user was seeing with respect to a display range that the user iscurrently seeing. Where a user wishes to know a location that has beenseen by another user, the user can move the display range in thedirection represented by the image 51 so as to see the same location asthe location which has been seen by the other user. Note that in theexample shown in FIG. 21, where the current display range (partially)overlaps a display range identified by history information, an image maybe displayed for notifying the user of such an overlap.

In FIG. 21, together with the image 51 representing history-relatedinformation, an image 52 representing user information associated withhistory-related information (user information stored while beingassociated with history information) is displayed. Thus, a user canidentify the user who was seeing the display range identified by thehistory-related information. Now, the panoramic video is a video whosearea cannot be contained within one screen, and the display range on theterminal device 2 can be freely controlled by the user. Therefore, sincethe position of the display range and the transition thereof differ fromone person (user) to another, history information of the display rangecan be said to reflect the personality of the user, and such historyinformation can be information that is useful for (interesting to) otherusers. Therefore, by presenting user information together with historyinformation, it is possible to improve the level of interest of a userto the panoramic video, and to motivate the user to view the panoramicvideo many times. Note that also in other display modes (FIGS. 22 and23) in which other contents are displayed as will be described later,user information may be displayed together with history-relatedinformation.

(History-Related Information Representing Display Frequency)

FIG. 22 shows another example of an image of history-relatedinformation. The history-related information shown in FIG. 22 presents,to a user, the display frequency for a predetermined range on apanoramic image obtained from the history information. In FIG. 22, animage 53 is displayed as history-related information, representing thenumber of times the current display range has been displayed in pastinstances of playback of the panoramic video. Note that the image usedfor presenting the display frequency to a user may by any image, otherthan the image 53, which represents the number of times. The informationprocessing device 3 may display, together with the panoramic video, userinformation (e.g., a character representing each user) for each positionand/or range obtained from the range information included in the historyinformation, for example. Thus, since a number of pieces of userinformation are displayed in accordance with the number of times thecurrent display range has been displayed in past instances of playbackof the panoramic video, the user can know how many or how few the numberof times is based on the number of pieces of user information displayed.

Where the number of times is calculated as the display frequency, theinformation processing system 1 first selects one of the rangeinformation included in the history information that is associated withthe time information corresponding to the current playback time point.Then, a range or position on a panoramic image is identified based onthe selected range information, and it is determined whether theidentified range or position is included within the current displayrange. The information processing system 1 makes the above determinationfor each piece of history information stored in the storage unit, andcounts the number of pieces of history information for which thedetermination result is affirmative. The image 53, which represents thecount obtained as described above, is displayed on the terminal device2. The history-related information of the image 53 represents statisticsobtained from a plurality of pieces of history information. The image 53is displayed at a predetermined position on the screen of the LCD 11.Note that the information processing system 1 may update the image 53repeatedly (e.g., every frame).

The number of times is the number of pieces of history information,among the one or more pieces of history information stored, that satisfythe condition regarding time information (the time informationcorresponding to the current playback time point) and the conditionregarding range information (the range or position represented by therange information being included within the current display range).Therefore, it is possible to calculate the display frequency over aplurality of instances of playback of the panoramic video, and it ispossible to present the display frequency to the user. That is, as theimage 53 is presented, a user can know how much the display range theuser is currently seeing has been seen by other users. For example, auser can play the panoramic video while setting the display range to arange that is popular (frequency seen by other users) or while settingthe display range to a range that is not frequency seen by other users.With the image 53, it is also possible to present, to the user,statistical information based on a plurality of users (or a plurality ofviews).

Note that the display frequency calculated from history information isnot limited to a display frequency calculated for a plurality ofinstances of playback of the panoramic video, but may also be a displayfrequency calculated for a single instance of playback of the panoramicvideo. That is, the display frequency may represent how much apredetermined range in a panoramic image has been seen during a singleinstance of playback of the panoramic video. FIG. 23 shows anotherexample of an image of history-related information based on historyinformation. The history-related information shown in FIG. 23 relates toan object represented by the object information described above, andrepresents how much the object (the range of the object) has been seen(displayed) in a single instance of playback of a panoramic video. Thatis, in FIG. 23, for an object within the current display range, an image54 representing the display frequency (degree of attention) of theobject determined based on history information is displayed ashistory-related information. Note that as shown in FIG. 23, the image 54representing history-related information is displayed while beingassociated with a position (range) of the object associated with thehistory-related information. As the image 54 is presented, a user canknow how much attention the object was drawing when other users playedthe panoramic video.

Where the display frequency is calculated for a single instance ofplayback of the panoramic video, the information processing system 1calculates the number of pieces of range information included in thehistory information that satisfy a predetermined condition regarding theposition on the panoramic image. Specifically, in the presentembodiment, the information processing system 1 determines whether theobject is included within the display range identified by the rangeinformation for each piece of range information included in the historyinformation. Note that this determination can be made based on eachpiece of range information and the object information. The informationprocessing system 1 counts the number of pieces of range information forwhich the determination result is affirmative, and determines the degreeof attention in accordance with the obtained count. For example, wherethe count is greater than or equal to a predetermined number, the image54 indicating that the degree of attention is high is output ashistory-related information while being associated with the object. Onthe other hand, where the count is smaller than a predetermined number,another image indicating that the degree of attention is low may beoutput as history-related information, or no history-related informationmay be output.

Note that in other embodiments, history-related information may becalculated representing the display frequency (degree of attention) ofan object over a plurality of instances of playback of the panoramicvideo. Then, the information processing system 1 performs the process ofcalculating the number of range information for each piece of historyinformation stored in the storage unit.

The display frequency represented by history-related information onlyneeds to represent the display frequency for a predetermined range in apanoramic image, and the predetermined range may be a display range or arange of an object.

The display frequency (degree of attention) may be determined using theplayback control information described above. For example, theinformation processing system 1 may determine that the degree ofattention for an object included within the display range is high whenthe user paused the playback. Alternatively, the display frequency (orthe degree of attention) may be determined to be high for a range thatwas displayed during a certain period that was rewound and playedrepeatedly, for example.

Note that history-related information may possibly be displayed at aparticular position on a panoramic image, such as the image 54, forexample. In such a case, the information processing system 1 may displayhistory-related information so as to move following the movement of theobject associated with the history-related information, as with theadditional information described in (3-3-2) above.

In other embodiments, the information processing system 1 may output asound representing history-related information, in addition to orinstead of displaying an image representing history-related information.Note that the sound is output from the terminal device 2 and/or themonitor 4.

(Output Destination of History-Related Information)

In the present embodiment, history-related information is also displayedon the terminal device 2, as with the additional information describedabove in [3. Inputting comment by user], for example. Thus, it ispossible to allow a user to more certainly recognize history-relatedinformation. Note that as with additional information, history-relatedinformation may also be displayed on the monitor 4. Thus, the panoramicvideo displayed on the terminal device 2 can be presented in aneasy-to-view manner. In other embodiments, the information processingsystem 1 may display an entire panoramic image on the monitor 4, anddisplay history-related information together with the entire panoramicimage (while being superimposed on the entire panoramic image).

(B) Method of Playing Display Range Determined Based on HistoryInformation

In the present embodiment, the information processing system 1 candetermine the display range based on history information, and play apanoramic video. Specifically, the information processing system 1determines a display range at each playback time point (each frame) of apanoramic video based on the range information stored in historyinformation. Then, a panoramic image of the determined display range isdisplayed on a display device. Thus, a user can play a panoramic videowith the same display range as that when history information was stored.

In the present embodiment, the panoramic video of a display rangedetermined based on history information is displayed on the monitor 4.On the other hand, a panoramic video of a display range determined basedon a user input is displayed on the terminal device 2. Therefore, a usercan see a panoramic video of an intended display range using theterminal device 2, and at the same time can see a panoramic video of adisplay range that has been seen by another user. Note that in otherembodiments, the information processing system 1 can play a panoramicvideo of a display range determined based on history information simplyon a single display device (the terminal device 2 or the monitor 4). Inother embodiments, where an entire panoramic image or an image of thespace represented by a panoramic image (e.g., a birds-eye view or a map)is displayed on the monitor 4, history-related information may bedisplayed together with the image displayed on the monitor 4 (whilebeing superimposed on the image).

FIG. 24 shows an example case where a panoramic video of a display rangedetermined based on history information is displayed on the monitor 4.In FIG. 24, a panoramic video 56 of a display range determined based onhistory information is displayed on the monitor 4, together with apanoramic video 55 of a display range such that the viewing direction ofthe virtual camera 22 is facing in the front direction. Thus, thepanoramic video 56 of the display range based on history information maybe displayed on a portion of the screen area of the monitor 4. Thepanoramic video 56 of the display range based on history information maybe displayed on a portion of the screen area of the terminal device 2.For example, a plurality of display ranges may be determined from aplurality of pieces of history information, and the panoramic video ofthe determined display ranges (a plurality of panoramic videos) may bedisplayed on the monitor 4 and/or the terminal device 2.

The display range may be determined based on a plurality of pieces ofhistory information. For example, the information processing system 1may identify a display range with a high display frequency using aplurality of pieces of history information, and display a panoramicvideo of the identified display range.

(C) Method of Displaying Another Image Together with Panoramic Video

In other embodiments, the information processing system 1 may produce anauxiliary image based on history information, and display the auxiliaryimage together with a panoramic video. An auxiliary image is an image inaccordance with the display range determined based on historyinformation and, in other words, is an image that can make a userrecognize the display range. An auxiliary image may be an image of a map(or a birds-eye view) of the space represented by the panoramic video asviewed from above, for example.

FIG. 25 shows an example case where an auxiliary image is displayedtogether with a panoramic video. In FIG. 25, a map image 62 of the spacerepresented by a panoramic video is displayed as an auxiliary image onthe terminal device 2, together with a panoramic video 61 of the currentdisplay range. The map image 62 is an image representing a map of thespace represented by the panoramic video 61 as viewed from above. Themap image 62 includes a viewpoint image 63, for example. The viewpointimage 63 represents the position of the viewpoint on the map. That is,the viewpoint image 63 representing a camera is arranged at the positionof the viewpoint on the map. Note that where the moving direction andthe moving velocity of the viewpoint are known in advance, the positionof the viewpoint on the map can be identified from the current playbacktime point (frame number).

The viewpoint image 63 represents the viewing direction on the map. Thatis, the viewpoint image 63 representing a camera is arranged so as toface in the viewing direction on the map. The viewing direction can becalculated from the display range determined based on historyinformation (e.g., using the attitude of the virtual camera 22). Thus,the information processing device 3 may display, on a display device, animage (the viewpoint image 63) representing the viewing directioncorresponding to the display range based on range information. Thus, itis possible to present, to a user, the direction in which another userwas looking. Note that while the viewpoint image 63 represents theviewing direction corresponding to the display range based on thehistory information in FIG. 25, it may represent the viewing directioncorresponding to a position based on the history information.

The map image 62 may include history-related information. For example,in FIG. 25, an image 64 indicating that the degree of attention is highis arranged on the map. Then, history-related information may or may notbe displayed also on the panoramic video 61. Thus, the informationprocessing device 3 may display, on a display device, an imagerepresenting the space represented by a panoramic video and a positionin the space corresponding to a position based on range information (aposition on the panoramic video. Herein, the position of history-relatedinformation). Thus, the position represented by range information can beexpressed by an image as viewed from a viewpoint different from thepanoramic video, and it is possible to present the position to a user inan easier-to-understand manner.

Note that the position on the map of history-related information can becalculated from the position on a panoramic video of the history-relatedinformation. The position on the map may be calculated by any method.For example, information in which positions on a panoramic video (orattitudes of the virtual camera 22) and positions on the map areassociated with each other may be provided in advance, so that aposition on the map can be calculated from a position on the panoramicvideo based on the information. For example, where object information isprovided, the position on the map for the object represented by theobject information may be calculated in advance, the object may beidentified from the position on the panoramic video, and the position onthe map can be identified from the object.

Thus, a display range and/or history-related information based onhistory information may be represented by using a map image. A displayrange based on history information and/or history-related informationmay be displayed on another auxiliary image different from the panoramicvideo, so that the panoramic video can be displayed in an easy-to-viewmanner.

In other embodiments, the information processing system 1 may play apanoramic video by simultaneously using some of the methods (A) to (C)described above. For example, the information processing system 1 maydisplay history-related information together with a panoramic video onthe terminal device 2, while displaying a panoramic video of a displayrange determined based on history information on the monitor 4.

(5-1-3) Obtaining History Information

In the first output mode in which the playback content from a pastinstance of playback is reflected, the device for obtaining historyinformation and the device for reflecting the history information may bethe same device or may be different devices. That is, the informationprocessing device 3 may obtain, as history information, rangeinformation regarding a panoramic video played on a display device thatdisplays the panoramic video reflecting the history information. Theinformation processing device 3 may obtain range information regarding apanoramic video played on another display device different from adisplay device that displays the panoramic video reflecting the historyinformation. Then, the information processing device 3 may play thepanoramic video using two display devices, as described in (B) above,for example. That is, while the panoramic video is displayed on thedisplay device (the terminal device 2 in the present embodiment), theinformation processing device 3 may determine the display range based onan input made on a predetermined input device and display, on the otherdisplay device (the monitor 4 in the present embodiment), the panoramicvideo of the determined display range (FIG. 24).

The information processing system 1 may obtain history information fromanother device outside the information processing system 1, in additionto storing history information in response to an input made on theterminal device 2. Then, a panoramic video may be output while changingthe output content thereof in accordance with the obtained historyinformation. For example, the information processing device 3 may becapable of communicating with an external device via a network such asthe Internet, in which case the information processing device 3 mayobtain history information from the external device. Thus, a user cansee, together with a panoramic video, information based on the historyinformation of another user (e.g., a friend or a celebrity).

Note that where history information is obtained from an external device,the history information may include user information representing theuser who has made the input represented by the history information.Then, the information processing system 1 may output user informationwhen outputting the panoramic video. For example, when history-relatedinformation is output, user information may be output together withhistory-related information (see FIG. 20), or when the display range isdetermined based on history information, user information may be outputtogether with a panoramic video of the determined display range.

(5-2) Second Output Mode

In the present embodiment, the information processing system 1 plays thesame panoramic video on two display devices (the terminal device 2 andthe monitor 4), where the playback content from one of the displaydevices is reflected in the playback of the panoramic video on the otherdisplay device (the fourth playback mode to be described below). Thatis, while the panoramic video is displayed on a display device (e.g.,the monitor 4), the information processing device 3 outputs thepanoramic video on another display device different from the displaydevice (e.g., the terminal device 2). Then, when the panoramic video isplayed on the other display device (the terminal device 2), theinformation processing device 3 obtains the range information. Moreover,while the panoramic video is played on the other display device (theterminal device 2), the information processing device 3 displays thepanoramic video on the display device (the monitor 4) while changing theoutput content thereof in accordance with the obtained rangeinformation.

Thus, where a plurality of users are simultaneously seeing a singlepanoramic video, it is possible to present, to a user, interestinginformation for the user, i.e., which locations in the panoramic videoare seen by other users, together with the panoramic video. The secondoutput mode in which the playback content from a concurrent instance ofplayback on another device is reflected will now be described in detail.

(5-2-1) Obtaining Range Information

In the second output mode, while a panoramic video is played, theinformation processing system 1 at least obtains range information. Inthe present embodiment, while a panoramic video is played on theterminal device 2 and the monitor 4, the information processing device 3obtains range information regarding the playback on the terminal device2. Note that in the second output mode, range information obtained fromthe playback on one display device is reflected in the playback onanother display device in real time, and therefore time information doesnot need to be obtained. Note however that where the output contentthereof is changed based on range information obtained in the past inaddition to range information obtained at the current point in time, theinformation processing device 3 may obtain time information togetherwith range information, as in the first output mode described above. Theinformation processing device 3 may obtain one or more of userinformation, zoom information, playback control information, etc.,together with range information.

(5-2-2) Playing Panoramic Video Using Range Information

The information processing system 1 plays a panoramic video based onrange information. That is, the information processing system 1 outputsa panoramic video while changing the output content (display mode)thereof in accordance with range information. Therefore, it is possiblenot only to simply present a panoramic video, but also to presentinformation (range information) regarding the playback content fromanother display device to the user. In the second output mode, themethod for outputting the panoramic video while changing the outputcontent thereof in accordance with the range information may be asfollows, for example.

In the present embodiment, the information processing device 3 plays, onthe monitor 4, a display range determined based on range informationobtained from the playback on the terminal device 2. That is, an imageof a panoramic video of the display range on the terminal device 2 isdisplayed in a partial area of the monitor 4. Specifically, while apanoramic video is displayed on the monitor 4, the informationprocessing device 3 determines a display range based on an input made ona predetermined input device (the terminal device 2), and displays thepanoramic video of the determined display range on the terminal device2. Then, the information processing device 3 displays a panoramic videoof a predetermined range on the monitor 4, while displaying a panoramicvideo of a display range based on the range information on the monitor4. Then, for the image displayed on the monitor 4, a panoramic video ofa display range based on range information may be displayed on a partialarea of the monitor 4, together with the panoramic video 55 of a displayrange such that the viewing direction of the virtual camera 22 is facingin the front direction, as in FIG. 24, for example.

Thus, it is possible to check the playback content of the terminaldevice 2 by watching the monitor 4. Now, since the terminal device 2 isof a portable type, it is difficult for a plurality of people to see thedisplay screen of the terminal device 2. In contrast, with the playbackcontent of the terminal device 2 displayed on the monitor 4, anotheruser other than the user of the terminal device 2 can easily check theplayback content of the terminal device 2. Thus, the user of theterminal device 2 and the other user can easily communicate with eachother, and the other user who does not control the display range, forexample, can also sufficiently enjoy the panoramic video.

Note that where a panoramic video whose display range is controlled onthe terminal device 2 is displayed on the monitor 4, another user willsee the panoramic video of the display range which the other user doesnot operate. Then, if the display range changes significantly, the otheruser may find it difficult to see the panoramic video. For example, ifthe display range changes abruptly, and in similar cases, the other usermay find it difficult to see the panoramic video. For example, where thedisplay range moves as the terminal device 2 itself is moved as in thepresent embodiment, if there are small movements (shaking) of thedisplay range due to shaking of a hand, the other user may find itdifficult to see the panoramic video. Therefore, the informationprocessing device 3 may display the panoramic video on the displaydevice (the monitor 4) while reducing the change in the display rangedetermined based on the range information. Therefore, it is possible topresent a panoramic video that is easier to view for other users. Notethat the method for reducing the change in the display range may be asfollows, for example.

For example, a monitor virtual camera is set in the virtual world havingthe same viewpoint and the same viewing direction as those of a terminaldevice virtual camera for displaying the panoramic video on the terminaldevice 2, and the monitor virtual camera is operated with basically thesame movement as the movement of the terminal device virtual camera. Inthe first control example, if the amount of change per unit time in theorientation of the terminal device 2 in the real space is less than apredetermined threshold value, or if a value (a value representing theacceleration or the angular velocity) output from a sensor (theacceleration sensor 14, the gyrosensor 15, etc.) for detecting themovement of the terminal device 2 is less than a predetermined thresholdvalue, and in similar cases, the attitude of the monitor virtual camerais fixed, thereby preventing the relatively small shaking of the imagedue to shaking of a hand. Then, when the amount of change or the outputvalue becomes zero or greater than or equal to the threshold value, theattitude of the monitor virtual camera is controlled so as to be closerto, or the same as, the attitude of the terminal device virtual camera.

In the second control example, if the amount of change per unit time inthe orientation of the terminal device 2 in the real space is greaterthan or equal to a predetermined threshold value, or if a value outputfrom a sensor for detecting the movement of the terminal device 2 isgreater than or equal to a predetermined threshold value, and in similarcases, the movement of the monitor virtual camera is restricted so as tosuppress relatively large changes in the viewing direction. Then, whenthe amount of change or the output value becomes less than the thresholdvalue, the attitude of the monitor virtual camera is controlled so as tobe closer to the attitude of the terminal device virtual camera.

In the third control example, the movement of the monitor virtual camerais controlled so as to follow the movement of the terminal devicevirtual camera at a predetermined rate, thereby suppressing therelatively small shaking of the image due to shaking of a hand or therelatively large changes in the viewing direction. Specifically, theviewing direction of the monitor virtual camera is controlled so as toalways change toward the viewing direction of the terminal devicevirtual camera at a predetermined rate (e.g., 50%) with respect to theangular difference between the viewing direction of the terminal devicevirtual camera and the viewing direction of the monitor virtual camera.

By displaying, on the monitor 4, the image of the inner surface of thethree-dimensional model described above as viewed from the monitorvirtual camera whose attitude is controlled using one of the controlexamples described above or a combination thereof, it is possible todisplay, on the monitor 4, an image obtained by reducing the change inthe viewing direction in the panoramic video displayed on the terminaldevice 2.

An example has been described above for reducing the change in themovement of the display range in a case where a panoramic video beingdisplayed on one display device is displayed on another display device.Note that in other embodiments, it is possible to reduce the change inthe display range when the display range is enlarged or shrunk. Forexample, when a user holding the terminal device 2 performs an operationof enlarging or shrinking a panoramic video being displayed on theterminal device 2, the panoramic video enlarged or shrunk in accordancewith the operation is displayed as it is on the LCD 11. On the otherhand, where a panoramic video which is displayed on the terminal device2, having been enlarged or shrunk in accordance with the operationperformed on the terminal device 2, is displayed also on the monitor 4,if the enlargement factor or the shrinkage factor of the panoramic videois greater than a predetermined threshold value, a panoramic video inwhich the enlargement factor or the shrinkage factor have beensuppressed to the threshold value is displayed on the monitor 4. Wherethe amount of time from when the panoramic video is enlarged to when itis shrunk in accordance with an operation performed on the terminaldevice 2, or the amount of time from when it is shrunk to when it isenlarged, is within a predetermined length of time, a panoramic video isdisplayed on the monitor 4 while suppressing the shrinking or theenlargement which has been done within the predetermined length of time.By displaying, on the monitor 4, a panoramic video in which theenlargement or the shrinkage is controlled by using one of the controlexamples or a combination thereof, it is possible to display, on themonitor 4, a panoramic video in which the change in the image enlargingor shrinking on the terminal device 2 is reduced.

In other embodiments, the information processing device 3 may display,on a display device, an image representing the viewing directioncorresponding to the display range or the position based on the rangeinformation, together with a panoramic video. FIG. 26 shows an exampleof an image representing the viewing direction. In FIG. 26, thepanoramic video of a display range corresponding to the viewingdirection in accordance with the attitude of the terminal device 2(herein, a viewing direction that is slightly to the right with respectto the front direction) is displayed on the terminal device 2. An image66 representing the viewing direction corresponding to the display rangeor the position based on the range information is displayed on themonitor 4, together with a panoramic video 65 of a display rangeobtained when the viewing direction of the virtual camera 22 is facingin the front direction. In the present embodiment, the image 66representing the viewing direction shows a character holding theterminal device looking like a user of the terminal device 2. Theattitude of the terminal device held by the character changes followingthe actual attitude of the terminal device 2. Since the attitude of theterminal device 2 corresponds to the viewing direction, it is possibleto notify the user of the viewing direction by displaying the image 66representing the attitude of the terminal device 2. Note that in FIG.26, the direction of the character represented by the image 66 iscontrolled so that it is equal to the direction of the user of theterminal device 2. In contrast, in other embodiments, the direction ofthe character may be controlled so that it is opposite in the front-backdirection to the direction of the user of the terminal device 2 (so thatit is a mirror image of the user of the terminal device 2) so that it iseasier to see the face of the character. In other embodiments, the mapimage 62 and the viewpoint image 63 may be displayed, as imagesrepresenting the viewing direction, together with the panoramic video.Also in this manner, it is possible to notify the user of the viewingdirection.

In other embodiments, while a panoramic video is played, the informationprocessing device 3 may obtain range information representing a positionspecified by a user, and display, on a display device, an imagerepresenting a position on a panoramic image determined based on therange information together with the panoramic video. For example, whenadditional information is input on a certain position on the panoramicvideo on the terminal device 2, the additional information may bedisplayed at the position also on the monitor 4. FIG. 27 shows anexample case where an image representing a position based on rangeinformation is displayed together with a panoramic image. In FIG. 27, itis assumed that an image 67 representing additional information has beeninput and displayed within the display range of the terminal device 2.Then, the information processing device 3 obtains range informationrepresenting the position at which the additional information has beeninput within the display range. Then, the information processing device3 displays an image 68 representing the additional information at theposition represented by the obtained range information in the displayrange displayed on the monitor 4. Thus, a user can check additionalinformation that has been input on the terminal device 2 also on themonitor 4, allowing another user other than the user of the terminaldevice 2 to check the additional information. Note that if the positionrepresented by the obtained range information is outside the displayrange on the monitor 4, the image 68 is not displayed on the monitor 4.In such a case, the information processing device 3 may display a guideimage described above on the monitor 4. The period over which the image68 is displayed is determined based on the period over which the image67 representing the additional information associated therewith isdisplayed on the terminal device 2. Therefore, the informationprocessing device 3 may obtain time information representing the periodin addition to obtaining range information representing the position atwhich the additional information has been input.

Note that the present embodiment is directed to an example where theplayback content of the terminal device 2 is reflected on the monitor 4.In other embodiments, the information processing device 3 may reflectthe playback content of the monitor 4 on the terminal device 2. Forexample, the information processing device 3 may display the panoramicimage of the display range on the monitor 4 on the terminal device 2.For example, where a position within the display range on the monitor 4can be specified by the controller device 7, the information processingdevice 3 can display, on the terminal device 2, an image representing aposition specified within the display range on the monitor 4.

As described above, in the first output mode and the second output mode,while the panoramic video is played, the information processing device 3obtains range information with which it is possible to identify adisplay range (and/or the position within the display range) that isdetermined based on an input made on a predetermined input device (theterminal device 2). Then, when a panoramic video is output by using adisplay device (the monitor 4), the panoramic video is output whilechanging the output content thereof in accordance with the obtainedrange information. Therefore, the playback content from past instancesof playback of the panoramic video or other instances of playback of thepanoramic video on other devices can be reflected in the output of thepanoramic video currently played, and it is possible to present moreinformation to a user while displaying a panoramic image.

(5-3) Output Based on History Information after Completion of Playback

Note that in the first output mode, range information is stored ashistory information. After the completion of playback of a panoramicvideo, the information processing device 3 may output information basedon the history information obtained during the playback. For example,after the completion of playback, the information processing device 3can present an object with a high display frequency during the playbackof the panoramic video or a display range with a high display frequencyover a plurality of instances of playback. Thus, it is possible topresent, to a user, interesting information, i.e., where the user wasseeing during the playback of the panoramic video.

For example, when the playback of a panoramic video is completed, theinformation processing device 3 may display, as a still image or avideo, a panoramic image of a display range that is determined based onthe range information stored as history information during the playbackon a display device (the terminal device 2 and/or the monitor 4). Anexample of presenting a panoramic image of a display range determinedbased on history information after the completion of playback of apanoramic video will now be described.

FIG. 28 shows an example of an image displayed after the completion ofplayback of a panoramic video. In FIG. 28, the panoramic image 68including an object with a high display frequency during the playback ofthe panoramic video is displayed on the monitor 4. Then, an image 69pointing at the object may be displayed. Note that the informationprocessing device 3 may display the panoramic image including the objectas a still image or as a video. Thus, the information processing device3 may display an object with a high display frequency during theplayback of a panoramic video (i.e., a frequency seen object) after thecompletion of playback.

Specifically, the information processing device 3 obtains the rangeinformation as history information (during the playback of a panoramicvideo). Then, the information processing device 3 determines whether anobject appearing in the panoramic video is included within the displayrange identified by the history information, and selects an object to beoutput in accordance with the determination result. Note that thedetermination may be made by using the history information and theobject information described above, for example. For example, theinformation processing device 3 may calculate, through the determinationdescribed above, the display time for each of a plurality of objectsappearing in the panoramic video, and select an object having a longdisplay time (high display frequency). After the completion of playbackof the panoramic video, the information processing device 3 displays thepanoramic image of the display range including the selected objecttherein on a display device (e.g., the monitor 4).

In the above example, the information processing device 3 may obtain, ashistory information, range information representing an object with ahigh display frequency during the playback of a panoramic video, andidentify the object represented by the history information after thecompletion of playback. Specifically, the information processing device3 determines (while a panoramic video is played) whether an object isincluded within the display range during the playback of the panoramicvideo, and selects an object to be output in accordance with thedetermination result. Note that as with the first method, thisdetermination can also be made by using the object information describedabove, for example. The information processing device 3 stores, as rangeinformation, information representing the selected object. Thus, whereit is possible to identify a display range and/or a position within thedisplay range on the panoramic video from information representing anobject, the information representing an object may be stored as rangeinformation. After the completion of playback of the panoramic video,the information processing device 3 displays the panoramic image of thedisplay range including the object represented by the range informationon a display device (e.g., the monitor 4).

Where there are a plurality of pieces of history information, theinformation processing device 3 may display, as a still image or avideo, the panoramic image of the display range determined based on aplurality of pieces of history information on a display device. Forexample, after the completion of playback of the panoramic video, theinformation processing device 3 identifies a range (or a position) witha high display frequency of the area of the panoramic video based on aplurality of pieces of history information. Then, the panoramic image ofthe identified display range is displayed on a display device (e.g., themonitor 4).

Note that in other embodiments, the information processing device 3 maydisplay panoramic images of a plurality of display ranges for objects ordisplay ranges in the descending order of display frequency. Then, auser can know the ranking of much-seen locations (objects) of thepanoramic video. The information processing device 3 may display apanoramic image of an object or a display range with a low displayfrequency (e.g., a display range including an object that has never beendisplayed). Then, it is possible to notify the user of a location in thepanoramic video that has been not so frequency seen.

As described in (5-1) to (5-3) above, in the present embodiment, theinformation processing system 1 changes the output content thereof inaccordance with the range information obtained during the playback of apanoramic video, and outputs at least a part of the panoramic video.Thus, it is possible to present, to a user, information of the playbackof the panoramic video (e.g., which display ranges the user was seeing,etc.).

[6. Specific Process Example of Present Embodiment]

Referring to FIGS. 29 to 33, specific processes and operations of theinformation processing system 1 according to the present embodiment willnow be described. FIG. 29 shows an example of data stored in a storageunit (the memory 6) of the information processing system 1 in thepresent embodiment. As shown in FIG. 29, the memory 6 of the informationprocessing device 3 stores an information processing program 70,panoramic video data 71, and process data 72. Note that other than datashown in FIG. 29, the memory 6 may store operation data obtained fromthe terminal device 2, images to be output to display devices, imagedata used for generating such images, etc.

The information processing program 70 is a program to be executed by acomputer (the CPU 5) of the information processing device 3. Note thatthe information processing program 70 is stored in a program storingunit (a storage device, a storage medium, or the like) that can beaccessed by the information processing device 3 and that is providedinside or outside the information processing device 3. A part or wholeof the information processing program 70 stored in the program storingunit is loaded and stored in the memory 6 at an appropriate point intime to be executed by the CPU 5. A part or whole of the informationprocessing program 70 may be pre-stored (as a library, for example) inthe information processing device for executing the informationprocessing program 70.

The panoramic video data 71 represents panoramic video data to be playedon the information processing system 1. The panoramic video data 71includes data of a plurality of panoramic images of a panoramic video.Note that data of the object information and/or data of the additionalinformation may be included in the panoramic video data 71.

The process data 72 is data used in an information process (FIGS. 30 to33) to be performed by the CPU 5. The process data 72 includes displayrange data 73, input information data 74, history information data 75,and range information data 76.

The display range data 73 represents a display range determined inaccordance with a user input. The display range data 73 may beinformation showing a position or range on a panoramic image, or may beinformation showing a vector representing the attitude of the virtualcamera 22. The input information data 74 is data representing the inputinformation, and for example includes the various information discussedin “(3-2) Storing additional information” (see FIG. 6). The historyinformation data 75 is data representing the history information, andfor example includes the various information discussed in “(5-1-1)Storing history information” (see FIG. 20). The range information data76 is data representing the range information obtained while a panoramicvideo is played.

Next, a detailed flow of the information process according to thepresent embodiment will be described. In the present embodiment, uponstart-up of the information processing system 1, the CPU 5 of theinformation processing device 3 initializes storage units such as thememory 6, and loads the information processing program 70 onto thememory 6 from the program storing unit. Then, the CPU 5 starts executingthe information processing program 70.

Now, in the present embodiment, the information processing system 1 canplay a panoramic video in four different playback modes as follows.

-   -   First playback mode: a playback mode in which inputs of        additional information are accepted and history information is        stored while a panoramic video is played.    -   Second playback mode: a playback mode in which additional        information, guide information and history-related information        are output while a panoramic video is played.    -   Third playback mode: a playback mode in which a panoramic video        is played while determining a display range based on history        information.    -   Fourth playback mode: a playback mode in which a panoramic video        is played on two display devices, wherein the playback content        of one display device is reflected in the playback on the other        display device.

Which one of the first to fourth playback modes is executed may bedetermined in any manner, and may be determined by a selection by auser, for example. Note that the information processing system 1 may becapable of executing only one of the playback modes. Referring to FIGS.30 to 33, the flow of a video playback process in each playback modewill now be described.

Note that the processes of steps in the flow charts shown in FIGS. 30 to33 are merely an example, and the order of steps may be changed or otherprocesses may be performed in addition to (or instead of) the processesof steps, as long as similar results are achieved. While it is assumedin the present embodiment that the processes of steps in the flow chartsare performed by the CPU 5, the processes of some steps in the flowcharts may be performed by a processor or a dedicated circuit other thanthe CPU 5.

(Playback Operation in First Playback Mode)

FIG. 30 is a flow chart showing an example of a flow of a video playbackprocess in the first mode to be performed by the CPU 5 of theinformation processing device 3 in the present embodiment. When thefirst playback mode is selected to be executed, the CPU 5 performs theprocess shown in FIG. 30.

First, in step S1, the CPU 5 determines a display range based on aninput made on the terminal device 2. The process of step S1 is performedby a method described above in “[2. Playback of panoramic video]”, forexample. Note that in step S1, the attitude of the terminal device 2 fordetermining a display range may be calculated in any manner. In thepresent embodiment, the CPU 5 obtains the detection result of theacceleration sensor 14 and/or the gyrosensor 15 from the terminal device2, and calculates the attitude of the terminal device 2 based on thedetection result. Data representing a display range determined based onthe attitude of the terminal device 2 is stored in the memory 6 as thedisplay range data 73.

In step S2, the CPU 5 obtains operation data from the terminal device 2,and determines whether there is an input of additional information(comment) based on the operation data. That is, the CPU 5 accepts aninput of additional information (comment). An input of additionalinformation is done by the method described above in “(3-1) Inputtingadditional information”, for example. Where the determination result ofstep S2 is affirmative, the process of step S3 is performed. On theother hand, where the determination result of step S2 is negative, theprocess of step S4 is performed, skipping the process of step S3.

In step S3, the CPU 5 stores the input information data 74 in the memory6. The input information is stored by the method described above in“(3-2) Storing additional information”. As a specific process, the CPU 5stores data representing the input information in the memory as theinput information data 74.

In step S4, the CPU 5 stores the history information. The historyinformation is stored by the method described above in “(5-1-1) Storinghistory information”, for example. As a specific process, the CPU 5reads out the history information data 75 stored in the memory 6, andadds the current time information and range information (display rangedata) to the history information represented by the history informationdata 75. Note that where the history information data 75 is not stored,data representing the current time information and range information(display range data) is stored in the memory as the history informationdata 75. The user information is added to the history information at anypoint in step S4 in the process loop of steps S1 to S6.

In step S5, the CPU 5 displays a panoramic image. In the presentembodiment, a panoramic image is displayed both on the terminal device 2and the monitor 4. Specifically, the CPU 5 reads out the panoramic videodata 71 and the display range data 73 from the memory 6 to produce apanoramic image of a display range determined in step S1, amongpanoramic images associated with the current playback time point andoutput the produced panoramic image to the terminal device 2. Amongpanoramic images associated with the current playback time point, apanoramic image of a predetermined display range is generated and outputto the monitor 4.

In step S6, the CPU 5 determines whether the playback of the panoramicvideo is ended. For example, the CPU 5 determines whether the panoramicvideo has been completely played to the end, and/or whether aninstruction to end the playback has been given by a user. If thedetermination result from step S6 is negative, the process of step S1 isperformed again. Thereafter, the series of processes of steps S1 to S6is performed repeatedly until it is determined in step S6 that theplayback should be ended. On the other hand, if the determination resultfrom step S6 is affirmative, the process of step S7 is performed.

In step S7, the CPU 5 selects an object to be presented after thecompletion of playback based on history information. The selection ofthe object is done by a method described above in “(5-3) Output based onhistory information after completion of playback”, for example. Notethat the process of selecting an object to be presented may be performedduring the playback of the panoramic video, as described above. In otherembodiments, the CPU 5 may perform a process of selecting a displayrange based on history information instead of the process of selectingan object (or in addition to the process of selecting an object).

In step S8, the CPU 5 displays a panoramic image of a display rangeincluding the selected object therein on a display device. The displayprocess in step S8 is performed by a method described above in “(5-3)Output based on history information after completion of playback”, forexample. After step S7, the CPU 5 ends the video playback process shownin FIG. 30.

Note that in other embodiments, in the video playback process in thefirst mode, only one of the process of accepting an input of additionalinformation (steps S2 and S3) and the process of storing historyinformation (step S4) may be performed. The process of outputting basedon history information (steps S7 and S8) does not need to be performed.

(Playback Operation in Second Playback Mode)

FIG. 31 is a flow chart showing an example of a flow of a video playbackprocess in a second mode to be performed by the CPU 5 of the informationprocessing device 3 in the present embodiment. When the second playbackmode is selected to be executed, the CPU 5 performs the process shown inFIG. 31.

First, in step S11, the CPU 5 obtains various information from anexternal device. Various information refers to input informationincluding additional information, and/or history information. The CPU 5obtains various information by a method described above in “(3-4)Additional information from other users” and “(5-1-3) Obtaining historyinformation”, and stores it in the memory 6 as new pieces of the inputinformation data 74 or the history information data 75.

In step S12, the CPU 5 sets a target. As described above in “(4-1)Setting target”, for example, the position associated with theadditional information set in the panoramic video to be played may beset as the target. The CPU 5 stores target information representing theset target. That is, the CPU 5 stores data representing targetinformation in the memory 6 by a method described above in “(4-2)Storing target information”.

In step S13, the CPU 5 determines a display range based on an input madeon the terminal device 2. The process of step S13 is the same as theprocess of step S1 described above.

In step S14, the CPU 5 produces guide information to be output. That is,the CPU 5 reads out the input information data 74 and data representingthe target information from the memory 6, and determines whether or notto output guide information based on the input information and thetarget information. This determination is made by a method describedabove in “(4-3) Method for determining output of guide information”. Ifit is determined to output guide information, the CPU 5 produces animage (the guide image 41 shown in FIG. 17) representing guideinformation (and user information) by a method described above in “(4-4)Manner of outputting guide information”, and determines the position onthe screen at which to display the image.

In step S15, the CPU 5 produces additional information to be output.That is, the CPU 5 reads out the input information data 74 from thememory 6, and determines whether or not to output additional informationbased on input information. This determination is made by a methoddescribed above in “(3-3-1) Method for determining output of additionalinformation”. If it is determined to output additional information, animage (e.g., the comment image 32) representing additional information(and user information) is produced by a method described above in“(3-3-2) Manner of outputting additional information” and “(3-3-3)Output content of additional information”, and the position on thescreen at which to display the image is determined.

In step S16, the CPU 5 produces history-related information to beoutput. That is, the CPU 5 reads out the history information data 75from the memory 6, and produces an image representing history-relatedinformation based on history information. The production of the image isdone by a method described above in “(A) Method of outputtinghistory-related information” of “(5-1-2) Playing panoramic video usinghistory information”. The CPU 5 also determines the position on thescreen at which to display the image. Note that in other embodiments, instep S16, an auxiliary image described above in “(C) Method ofdisplaying another image together with panoramic video” and “(5-1-2)Playing panoramic video using history information” may be produced.Then, in step S17, an auxiliary image is displayed together with thepanoramic image.

In step S17, the CPU 5 displays images of the guide information, theadditional information and the history-related information together withthe panoramic image. Specifically, the CPU 5 reads out the panoramicvideo data 71 and the display range data 73 from the memory 6, andproduces a panoramic image of a display range determined in step S13, ofthe panoramic image associated with the current playback time point.Then, the images produced in steps S14 to S16 are superimposed on theproduced panoramic image, and the produced image is output to theterminal device 2. Note that in step S17, the process of displaying apanoramic video on the monitor 4 is the same as step S5 described above.

In step S18, the CPU 5 determines whether or not to end the playback ofthe panoramic video by a method similar to step S6. If the determinationresult from step S18 is negative, the process of step S13 is performedagain. On the other hand, if the determination result from step S18 isaffirmative, the CPU 5 ends the video playback process shown in FIG. 31.Thereafter, the series of processes of steps S13 to S18 is performedrepeatedly until it is determined in step S18 that the playback shouldbe ended.

Note that in other embodiments, in the video playback process in thesecond mode, only some of the various information (additionalinformation, guide information, and history-related information) may beoutput. For example, in the video playback process in the second mode,only the process of outputting additional information (step S14), onlythe processes of outputting additional information and guide information(steps S14 and S15), or only the process of outputting history-relatedinformation (step S16) may be performed, of the processes of outputtingvarious information (steps S14 to S16).

(Playback Operation in Third Playback Mode)

FIG. 32 is a flow chart showing an example of a flow of a video playbackprocess in the third mode to be performed by the CPU 5 of theinformation processing device 3 in the present embodiment. When thethird playback mode is selected to be executed, the CPU 5 performs theprocess shown in FIG. 32.

First, in step S21, the CPU 5 selects history information to be used fordetermining a display range. The history information may be selected byany method, and is selected following a user instruction, for example.

In step S22, the CPU 5 determines a display range based on an input madeon the terminal device 2. The display range determined in step S22 isthe display range of the panoramic video to be displayed on the terminaldevice 2. The process of step S22 is the same as the process of step S1described above.

In step S23, the CPU 5 determines a display range based on historyinformation selected in step S21. The display range determined in stepS22 is the display range of the panoramic video displayed on the monitor4. That is, the CPU 5 reads out the history information data 75representing the selected history information from the memory 6, anddetermines the display range by a method described above in “(5-2-2)Playing panoramic video using range information”. The CPU 5 stores datarepresenting the determined display range in the memory 6 as the displayrange data 73.

In step S24, the CPU 5 displays a panoramic image. Specifically, the CPU5 reads out the panoramic video data 71 and the display range data 73from the memory 6, and produces a panoramic image of a display rangedetermined in step S23, of the panoramic image associated with thecurrent playback time point, to output the panoramic image to themonitor 4. Then, the panoramic image of the display range based onhistory information may be displayed in a part of the screen area of themonitor 4 as shown in FIG. 24. Note that the process of displaying apanoramic video on the terminal device 2 in step S24 is the same as stepS5 described above.

Note that in other embodiments, in the video playback process in thethird mode, some of the process of accepting an input of additionalinformation (steps S2 and S3) and the process of storing historyinformation (step S4) may be performed. In the video playback process inthe third mode, some of the processes of outputting various informationsuch as additional information (steps S14 to S16) may be performed.

(Playback Operation in Fourth Playback Mode)

FIG. 33 is a flow chart showing an example of a flow of a video playbackprocess in the fourth mode to be performed by the CPU 5 of theinformation processing device 3 in the present embodiment. When thefourth playback mode is selected to be executed, the CPU 5 performs theprocess shown in FIG. 33.

First, in step S31, the CPU 5 determines a display range based on aninput made on the terminal device 2. The process of step S31 is the sameas the process of step S1 described above.

In step S32, the CPU 5 obtains range information. For example, the CPU 5obtains range information representing the display range determined instep S31 described above. The range information may be obtained by amethod described above in “(5-2-1) Obtaining range information”, forexample. Data representing the obtained range information is stored inthe memory 6 as the range information data 76.

In step S33, the CPU 5 produces an image based on the range information(an image reflecting the playback content of the terminal device 2) tobe output together with the panoramic video. That is, the CPU 5 readsout the range information data 76 from the memory 6, and produces theimage based on the range information. For example, an image describedabove in “(5-2-2) Playing panoramic video using range information” isproduced.

In step S34, the CPU 5 displays a panoramic video on the terminal device2. Note that the process of displaying a panoramic video on the terminaldevice 2 is the same as step S5 described above.

In step S35, the CPU 5 displays a panoramic video on the monitor 4 whilechanging the output content thereof in accordance with the rangeinformation. Specifically, the CPU 5 displays the image produced in stepS33 together with the panoramic image on the monitor 4. Thus, an imagereflecting the playback content of the terminal device 2 described abovein “(5-2-1) Obtaining range information” is displayed on the monitor 4.Note that the method of producing and displaying a panoramic image instep S35 is the same as step S5 described above.

In step S36, the CPU 5 determines whether or not to end the playback ofthe panoramic video by a method similar to step S6. If the determinationresult from step S36 is negative, the process of step S31 is performedagain. On the other hand, if the determination result from step S36 isaffirmative, the CPU 5 ends the video playback process shown in FIG. 33.Thereafter, the series of processes of steps S31 to S36 is performedrepeatedly until it is determined in step S36 that the playback shouldbe ended.

Note that in other embodiments, some processes of the first to fourthplayback modes may be performed at the same time (in one playback mode).For example, in the fourth playback mode, the process of inputtingvarious information of the first playback mode (steps S2 to S4) may beperformed, or the process of outputting various information of thesecond playback mode (steps S13 to S16) may be performed. Processes tobe performed by the information processing device 3 are as describedabove.

The systems, devices and apparatuses described herein may include one ormore processors, which may be located in one place or distributed in avariety of places communicating via one or more networks. Suchprocessor(s) can, for example, use conventional 3D graphicstransformations, virtual camera and other techniques to provideappropriate images for display. By way of example and withoutlimitation, the processors can be any of: a processor that is part of oris a separate component co-located with the stationary display and whichcommunicates remotely (e.g., wirelessly) with the movable display; or aprocessor that is part of or is a separate component co-located with themovable display and communicates remotely (e.g., wirelessly) with thestationary display or associated equipment; or a distributed processingarrangement some of which is contained within the movable displayhousing and some of which is co-located with the stationary display, thedistributed portions communicating together via a connection such as awireless or wired network; or a processor(s) located remotely (e.g., inthe cloud) from both the stationary and movable displays andcommunicating with each of them via one or more network connections; orany combination or variation of the above.

The processors can be implemented using one or more general-purposeprocessors, one or more specialized graphics processors, or combinationsof these. These may be supplemented by specifically-designed ASICs(application specific integrated circuits) and/or logic circuitry. Inthe case of a distributed processor architecture or arrangement,appropriate data exchange and transmission protocols are used to providelow latency and maintain interactivity, as will be understood by thoseskilled in the art.

Similarly, program instructions, data and other information forimplementing the systems and methods described herein may be stored inone or more on-board and/or removable memory devices. Multiple memorydevices may be part of the same device or different devices, which areco-located or remotely located with respect to each other.

As described above, the present embodiment is applicable to aninformation processing system or an information processing program, withthe aim of, for example, presenting more information to the user whendisplaying a panoramic image.

While certain example systems, methods, devices and apparatuses havebeen described herein, it is to be understood that the appended claimsare not to be limited to the systems, methods, devices and apparatusesdisclosed, but on the contrary, are intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims.

What is claimed is:
 1. A non-transitory computer-readable storage mediumcomprising an information processing program to be executed by acomputer of an information processing device for displaying a partialarea of a panoramic video on a display device, wherein the informationprocessing program instructs the computer to: obtain range informationdetermined based on an input made on a predetermined input device whilethe panoramic video is played, the range information indicating historyof a display range of the panoramic video at a point in time thepanoramic video was viewed by a user, wherein it is possible with therange information to identify the display range of the panoramic videoto be displayed and/or to identify a position within the display range;and output at least a part of the panoramic video while changing outputcontent thereof in accordance with the obtained range information thatindicates the history of the display range of the panoramic video at thepoint in time the panoramic video was viewed by the user.
 2. The storagemedium according to claim 1, wherein the obtained range information isstored in a storage device as history information indicating the historyof the display range of the panoramic video viewed by the user; andinformation based on the history information is output after the historyinformation is stored.
 3. The non-transitory computer-readable storagemedium according to claim 2, wherein the history information stored inthe storage device is information in which time information, related tothe point in time the panoramic video was viewed by the user, isassociated with range information, wherein it is possible with the timeinformation to identify a playback time point in the panoramic video,and wherein it is possible with the range information to identify adisplay range at the playback time point and/or a position within thedisplay range.
 4. The non-transitory computer-readable storage mediumaccording to claim 3, wherein the information processing program furtherinstructs the computer to determine the display range based on a userinput; and a panoramic video of a display range determined based on theuser input is displayed on the display device, and history-relatedinformation produced based on the history information is output.
 5. Thenon-transitory computer-readable storage medium according to claim 4,wherein the history-related information output together with thepanoramic video represents a direction of the display range identifiedby the range information included in the history information withrespect to a current display range.
 6. The non-transitorycomputer-readable storage medium according to claim 4, wherein theoutput history-related information represents a display frequency for apredetermined range in the panoramic video obtained from the historyinformation.
 7. The non-transitory computer-readable storage mediumaccording to claim 6, wherein the history-related information representsa display frequency determined in accordance with the number of piecesof history information that satisfy a predetermined condition regardingthe time information and the range information.
 8. The non-transitorycomputer-readable storage medium according to claim 6, wherein thehistory-related information represents a display frequency determined inaccordance with the number of pieces of range information that satisfy apredetermined condition regarding a position on a panoramic image. 9.The non-transitory computer-readable storage medium according to claim4, wherein the panoramic video is stored in the storage device togetherwith object information representing a position and/or a range of anobject appearing in the panoramic video; and history-related informationoutput together with the panoramic video represents a display frequencyof the object calculated based on the object information and the historyinformation.
 10. The non-transitory computer-readable storage mediumaccording to claim 2, wherein when playback of the panoramic video iscompleted, a panoramic image of a display range determined based on therange information stored as the history information during the playbackis displayed on the display device as a still image or a video.
 11. Thenon-transitory computer-readable storage medium according to claim 10,wherein range information representing the display range and/or aposition within the display range is obtained as the historyinformation; the information processing program further instructs thecomputer to determine whether or not an object appearing in thepanoramic video is included in a display range identified by the historyinformation so as to select an object to be output based on a result ofthe determination; and a panoramic image including the selected objecttherein is displayed on the display device as a still image or a video.12. The non-transitory computer-readable storage medium according toclaim 10, wherein the information processing program further instructsthe computer to determine whether or not an object is included within adisplay range during playback of the panoramic video so as to select anobject to be output based on a result of the determination; informationrepresenting the selected object is obtained as the range information;and a panoramic image of a display range including therein the objectrepresented by the range information is displayed on the display deviceas a still image or a video.
 13. The non-transitory computer-readablestorage medium according to claim 10, wherein the history informationstored in the storage device is information in which time information,related to the point in time the panoramic video was viewed by the user,is associated with range information, wherein it is possible with thetime information to identify a playback time point in the panoramicvideo, and wherein it is possible with the range information to identifya display range at the playback time point and/or a position within thedisplay range; and a panoramic image of a display range determined basedon a plurality of pieces of history information is displayed on thedisplay device as a still image or a video.
 14. The non-transitorycomputer-readable storage medium according to claim 2, wherein rangeinformation regarding a panoramic video played on the display device isobtained.
 15. The non-transitory computer-readable storage mediumaccording to claim 2, wherein range information regarding a panoramicvideo played on another display device different from the display deviceis obtained.
 16. The non-transitory computer-readable storage mediumaccording to claim 15, wherein the information processing programfurther instructs the computer to: determine the display range based onan input made on the predetermined input device while the panoramicvideo is displayed on the display device; and display the panoramicvideo of the determined display range on the other display device whilethe panoramic video is displayed on the display device.
 17. Thenon-transitory computer-readable storage medium according to claim 1,wherein a panoramic video of a display range determined based on therange information is displayed on the display device.
 18. Thenon-transitory computer-readable storage medium according to claim 1,wherein an image representing a viewing direction corresponding to adisplay range or a position based on the range information is displayedon the display device.
 19. The non-transitory computer-readable storagemedium according to claim 1, wherein the range information is obtainedfrom an external device via a network.
 20. The non-transitorycomputer-readable storage medium according to claim 1, wherein: userinformation representing a user who has made an input regarding therange information is further obtained; and the user information isoutput when the panoramic video is output.
 21. The non-transitorycomputer-readable storage medium according to claim 1, wherein the rangeinformation is capable of identifying at least a spatial display rangeof the panoramic video to be displayed and/or to identify a spatialposition within the display range.
 22. The non-transitorycomputer-readable storage medium according to claim 1, wherein thehistory includes time information related to the point in time thepanoramic video was viewed by the user, the time informationrepresenting a frame number of a panoramic image in the panoramic video.23. A non-transitory computer-readable storage medium comprising aninformation processing program to be executed by a computer of aninformation processing device for displaying a partial area of apanoramic video on a display device, wherein the information processingprogram instructs the computer to: obtain range information, regarding apanoramic video played on another display device different from thedisplay device, determined based on an input made on a predeterminedinput device while the panoramic video is played, the range informationindicating history of a display range of the panoramic video viewed by auser, wherein it is possible with the range information to identify thedisplay range of the panoramic video to be displayed and/or to identifya position within the display range; and output at least a part of thepanoramic video while changing output content thereof in accordance withthe obtained range information, the panoramic video is displayed on thedisplay device while reducing a change in the display range determinedbased on the range information.
 24. A non-transitory computer-readablestorage medium comprising an information processing program to beexecuted by a computer of an information processing device fordisplaying a partial area of a panoramic video on a display device,wherein the information processing program instructs the computer to:obtain range information representing a position specified by a userwhile the panoramic video is played and determined based on an inputmade on a predetermined input device while the panoramic video isplayed, the range information indicating history of a display range ofthe panoramic video viewed by a user, wherein it is possible with therange information to identify the display range of the panoramic videoto be displayed and/or to identify a position within the display range;and output at least a part of the panoramic video while changing outputcontent thereof in accordance with the obtained range information, animage representing a position on a panoramic image determined based onthe range information is displayed on the display device together withthe panoramic video.
 25. A non-transitory computer-readable storagemedium comprising an information processing program to be executed by acomputer of an information processing device for displaying a partialarea of a panoramic video on a display device, wherein the informationprocessing program instructs the computer to: obtain range informationdetermined based on an input made on a predetermined input device whilethe panoramic video is played, the range information indicating historyof a display range of the panoramic video viewed by a user, wherein itis possible with the range information to identify the display range ofthe panoramic video to be displayed and/or to identify a position withinthe display range; and output at least a part of the panoramic videowhile changing output content thereof in accordance with the obtainedrange information, an image representing a space represented by thepanoramic video and a position in the space corresponding to a positionon the panoramic video based on the range information is displayed onthe display device.
 26. An information processing device comprising atleast one processor and configured to display a partial area of apanoramic video on a display device, the at least one processorcontrolling the information processing device to: obtain rangeinformation determined based on an input made on a predetermined inputdevice while the panoramic video is played, the range informationindicating history of a display range of the panoramic video at a pointin time the panoramic video was viewed by a user, wherein it is possiblewith the range information to identify the display range of thepanoramic video to be displayed and/or to identify a position within thedisplay range; and output, where the panoramic video is output using thedisplay device, the panoramic video while changing output contentthereof in accordance with the obtained range information that indicatesthe history of the display range of the panoramic video at the point intime the panoramic video was viewed by the user.
 27. An informationprocessing system for displaying a partial area of a panoramic video ona display device, the information processing system comprising: aprocessing system having at least one processor, the processing systemconfigured to: obtain range information determined based on an inputmade on a predetermined input device while the panoramic video isplayed, the range information indicating history of a display range ofthe panoramic video at a point in time the panoramic video was viewed bya user, wherein it is possible with the range information to identifythe display range of the panoramic video to be displayed and/or toidentify a position within the display range; and output, where thepanoramic video is output using the display device, the panoramic videowhile changing output content thereof in accordance with the obtainedrange information that indicates the history of the display range of thepanoramic video at the point in time the panoramic video was viewed bythe user.
 28. A panoramic video display method for displaying a partialarea of a panoramic video on a display device, the panoramic videodisplay method comprising: obtaining range information determined basedon an input made on a predetermined input device while the panoramicvideo is played, the range information indicating history of a displayrange of the panoramic video at a point in time the panoramic video wasviewed by a user, wherein it is possible with the range information toidentify the display range of the panoramic video to be displayed and/orto identify a position within the display range; and outputting, wherethe panoramic video is output using the display device, the panoramicvideo while changing output content thereof in accordance with theobtained range information that indicates the history of the displayrange of the panoramic video at the point in time the panoramic videowas viewed by the user.
 29. A non-transitory computer-readable storagemedium comprising an information processing program to be executed by acomputer of an information processing device for displaying a partialarea of a panoramic video on a display device, wherein the informationprocessing program instructs the computer to: obtain range informationdetermined based on an input made on a predetermined input device whilethe panoramic video is played, the range information indicating historyof a display range of the panoramic video viewed by a user, wherein itis possible with the range information to identify the display range ofthe panoramic video to be displayed and/or to identify a position withinthe display range; and output at least a part of the panoramic videowhile changing output content thereof in accordance with the obtainedrange information, wherein the display range controlled by a user whilethe panoramic video is played is stored as history informationindicating the history of the display range of the panoramic videoviewed by the user, additional information is generated based on thehistory information, and the additional information is output whileoutputting at least the part of the panoramic video.